Adapting Pathway Programs to the Virtual World: Insights from the Chicago EYES on Cancer Response to COVID-19-Related Disruptions to Training

,


INTRODUCTION
Since spring 2020, coronavirus disease 2019  has disrupted development of the next generation of cancer researchers and physicians, forcing pathway programs nationwide to cancel, postpone or reinvent all education and training activities.Chicago EYES (Educators and Youth Enjoy Science) on Cancer, a cancer research training program for underrepresented high school and undergraduate students, was no exception.COVID-19-related campus closures rendered impossible the immersive laboratory experiences, face-to-face mentoring and peer interactions fundamental to the EYES program model.In response, the leadership team provided remote research experiences and online skill-building and career development activities, with the goal of preserving trainees' access to intellectual, social and financial support despite the pandemic.
Presented here are insights from the EYES program's intensive summer session, June through August 2020.First, we describe the leadership team's responses to COVID-19-re-lated programming challenges, which prioritized flexibility to accommodate trainees' diverse personal circumstances.Next, in Study 1, we examine trainee engagement and barriers to participation to assess accessibility of the redesigned program.Finally, in Study 2, we evaluate the impact of our online programming on trainees' research skills and career development relative to on-site, in-person programming offered to prior cohorts.The outcomes of our online EYES programming have not only informed our efforts to provide impactful cancer research training to underserved youth throughout the COVID-19 pandemic, but they have also inspired plans to extend online enrichment opportunities to program alumni, partner schools and other priority groups as a permanent component of the Comprehensive Cancer Center's broader cancer education strategy.We share our findings and lessons learned as a resource for others similarly focused on pursuing robust ways to serve promising young scientists.

CHICAGO EYES ON CANCER: BEFORE AND DURING COVID-19
Chicago EYES on Cancer is a cancer research training program for high school and undergraduate students interested in careers in biomedicine.It was initiated in 2017 through a National Cancer Institute (NCI) Youth Enjoy Science (YES) Research Education Program (R25) grant to the University of Chicago Medicine Comprehensive Cancer Center, currently one of 16 active YES grants nationwide (National Cancer Institute, 2020;National Institutes of Health, 2016).The R25 YES program is specifically designed to increase diversity within the biomedical research workforce, the need for which is well documented (National Academy of Sciences et al., 2011).Differing skillsets and perspectives within research teams is associated with higher-quality research, greater innovation and better health outcomes for under-resourced populations (Institute of Medicine and National Research Council, 2006; National Academies of Sciences and Engineering, 2019;Smith-Doerr et al., 2017).Yet, underrepresentation of those who identify as racial or ethnic minorities, economically disadvantaged, first-generation college and/or living with a disability persists in health-related sciences on a national scale (Association of American Medical Colleges, 2019; Meyers et al., 2018), contributing to devastating health disparities, including with regard to cancer (Cooper et al., 2003;Penner et al., 2016;Persky et al., 2013).
Consistent with priorities established by NCI, Chicago EYES on Cancer strives to meet three core objectives: (1) Equip young people with specialized research skills and expertise in biomedicine; (2) Broaden young people's awareness of career opportunities in biomedicine and empower them to make informed, strategic plans to accomplish career goals; and (3) Strengthen support for young people's career development by engaging members of the scientific and local communities, and young people's immediate families, in their education and training.To accomplish these goals, EYES typically comprises two consecutive, immersive summer research experiences under the mentorship of UChicago cancer research faculty (35 hours per week for 8 weeks each summer).Trainees also participate in year-round skill-building and career development activities facilitated by the EYES leadership team (5-10 hours per month for 12 months), including faculty lecture series, journal clubs and career talks, book clubs, and workshops on such topics as building a professional network and ethics in biomedicine.Finally, trainees collaborate on outreach projects for Chicagoland communities, such as the development of a cancer-themed series for Chicago public television.Trainees receive a yearly stipend, disseminated in several installments to correspond with trainees' completion of major program requirements.
The program is competitive, admitting approximately a dozen trainees each year from applicant pools consistently topping 200 (322 young people applied in 2020).Forty-six young people have participated in the program to date, all but one of whom identify as underrepresented in the sciences.Trainee demographics are reported in Table 1, under Program Overall.
Until March 2020, EYES programming was almost exclusively in person and held in UChicago laboratories and meeting spaces, part of a deliberate effort to expose trainees to the culture, sights and sounds of a professional research environment while fostering strong interpersonal relationships.When COVID prompted a campus-wide shut down to all but essential employees, the EYES leadership team worked quickly to adapt the program to a completely virtual format.Initial planning sessions with trainees revealed considerable diversity with regard to their personal circumstances.Some had no formalized plans for the summer apart from EYES, while others felt pressured to make new commitments to coursework, caretaking and part-or full-time employment.Trainees also differed in their access to appro-  priate electronic devices, reliable internet and private, quiet space to complete their work.Finally, trainees had varying degrees of research experience and expertise, and consequently, varied capacity for the particular demands of remote research.It became clear that, in order to make programming equitably accessible, distinct program tracks, activity choice and make-up work options would be key facets of the EYES redesign.
Trainees were organized into two distinct tracks based on their level of research experience.The Research track was essentially an online equivalent to the planned summer research experience: Trainees participated in 35 hours of cancer-related research each week under the mentorship of a Comprehensive Cancer Center faculty member.Most projects were computational in nature, for example, image analysis of clinical and radiological features of breast cancer to predict residual cancer burden; modeling of publicly available data to identify effective chemotherapeutic drugs for metastatic breast cancer; and a genome-wide association study to identify clinical and genetic variables contributing to ototoxicity in childhood cancer survivors following radiation to the brain.Other projects included a public health-oriented study of tobacco use and cessation related to services provided by health professionals.Trainees' research activities also included remote attendance at research team meetings, journal clubs and other events as requested by the mentor.In addition, trainees were required to attend weekly debrief meetings with EYES leadership, attend weekly EYES-sponsored journal clubs, and participate in a selection of skill-building and career development activities as described below.Finally, trainees presented their research projects during an end-of-summer virtual research symposium in the form of 10-minute PowerPoint presentations via Zoom webinar.
The Research track was reserved for second-year trainees, who the previous summer had acquired the level of skill, expertise and familiarity with the program itself to succeed despite the limited support and oversight that the circumstances allowed.One first-year trainee, a UChicago undergraduate, was also permitted to join the Research track, given her established relationship with a faculty mentor willing to oversee her project.These individuals will thus complete EYES program components according to the original sequence and timeline.
The Enrichment track engaged trainees in online activities to foster research skills and career development.This track required a part-time commitment (5-10 hours per week) and was designed to replace one cycle of required academic year activities.Mentored research experiences for this group were postponed until summer 2021.Thus, trainees in the Enrichment track will complete program components in an altered sequence and according to an extended timeline.All first-year trainees were encouraged to join the Enrichment track, as well as second-year trainees who, for reasons of availability or personal preference, opted out of the Research track.
Prior to the summer, trainees in both the Research and Enrichment tracks were presented with a detailed activity packet containing descriptions of 32 unique options, both synchronous and asynchronous, related to one of four themes: Concepts in Cancer, Research and Medicine; Research and Professional Skills; Careers in Biomedicine; and Science Outreach and Community Engagement.Many activities were simply the online version of those held in person during a typical summer session; for example, faculty hosts for the weekly journal club engaged trainees in discussions of the cancer research literature on Zoom rather than in a campus classroom.Other activities took advantage of opportunities uniquely available in the virtual world.In particular, nearly half of EYES trainees attended the American Association for Cancer Research (AACR) virtual meeting on COVID-19 and cancer, after a preliminary workshop on the purpose of professional meetings and best practice for engaging scientific colleagues.Throughout the three-day conference, trainees attend a selection of assigned sessions plus a few of their own choosing.They exchanged comments, questions and session recommendations in real time via the program's Microsoft Teams page, with live feedback from EYES leadership.Afterward, trainees participated in a group debrief session to discuss meeting highlights and lingering questions.It was the first time most trainees had attended a professional conference, and for many, a highlight of the summer.
The complete list of activity options is presented in Table 2.The activity packet distributed to trainees is available as a supplement to this publication.Each activity was allotted a point value based on the estimated time and effort required to complete it (1 point = roughly 5 hours; allotments ranged from .5 to 2 points).To secure "credit" for the summer component of the EYES program (and the corresponding portion of the participant stipend) trainees in the Research track were required to earn a minimum of 3 points beyond the research-related activities described above.Trainees in the Enrichment track were required to earn a minimum of 10 points.Trainees in both tracks chose freely those activities best aligned with their interests and availability.Activities such as workshops, journal clubs, panel discussions and faculty career talks were recorded and made accessible for viewing at trainees' convenience, in the event they could not attend the live sessions.
A final component of the redesigned EYES program was a series of virtual, non-credit social activities to foster a sense of support and community among trainees.Each member of the leadership team hosted weekly office hours open to any trainee interested in discussing program activities, career and personal interests.Trainees also met regularly with their assigned peer mentor: UChicago undergraduates, graduate students and postdoctoral fellows in the biomedical sciences.Formal social activities included a program orientation session, with time reserved for small group interactions in breakout rooms.A family night was scheduled midway through the summer, giving trainees' parents, siblings and other members of their support networks the opportunity to learn about the program and afterward, participate in a series of games guided by professional hosts.Each family was issued a $50 gift card for a meal to enjoy during the event.

STUDY 1: TRAINEE ENGAGEMENT AND BARRIERS TO PARTICIPATION, SUMMER 2020
By converting the EYES program to a completely virtual format -and by establishing flexibility as a central tenet of the redesign -the leadership team hoped to preserve trainees' access to skill-building experiences, social networks, and monetary award for participation.Study 1 examines the program's accessibility, as evidenced by aspects of trainee engagement and reported barriers to participation.
Methods.Analyses focused on data from the 26 trainees (12 second years and 14 first years) eligible to participate in EYES in summer 2020.Sample demographics are reported in Table 1, under Study 1. Engagement measures included the number of trainees to enroll in either the Research or Enrichment tracks, as well as the tally of points earned by each group for activities completed over the course of the summer.To assess barriers to participation, all trainees were administered, via REDCap, a series of items as part of the end-of-summer survey assessment ("post" assessment; response rate = 100%).Trainees rated on a scale from 1 (not at all) to 5 (a great deal) the extent to which four basic factors interfered with their participation: (1) access to a suit-able electronic device, (2) reliable internet, (3) conflicting responsibilities, and (4) emotional burden.A rating of 4 or 5 was interpreted as having a negative impact on the trainee's ability to participate.Two subsequent open-response items at post assessment invited trainees to describe ways in which the EYES leadership team did mitigate and could have mitigated these challenges, respectively.Common themes were identified through inductive coding.As a measure of program quality, additional open-response items at post invited trainees to identify which activities were most successful in supporting their growth in four key domains: scientific skills and expertise, professional skills, career knowledge and ambitions, and sense of support and belonging.
Frequencies and percentages were calculated for all categorical variables.
Results.Of the 26 trainees eligible to participate in 2020 summer programming, only 3 opted out, citing competing work demands, family obligations, or health concerns.All three individuals were second-year students, who will resume their EYES training by summer 2021.Of the 23 trainees who did participate, 7 (6 second years and 1 first year) committed to the Research track and 16 (3 second years and 13 first years) to the Enrichment track.
All participants achieved the minimum point requirements for activity completion and most (87%) exceeded them (required points for Research track = 3, median earned points = 7, range = 4 -14.5;required points for Enrichment track = 10, median earned points = 12.5, range = 10 -28).Of the 275.5 points awarded across participants, only 1 was for completion of an asynchronous activity, indicating a strong preference among trainees for synchronous forms of engagement.
Seventy-five percent of participants indicated at least one factor interfering with their ability to participate in the program (a rating of 4 or 5), and 25.0% indicated three or more  more programming focused on wellbeing in the future, specifically workshops on mental health.We did not anticipate trainees' overwhelming preference for synchronous activities, thinking instead that they would be experiencing "Zoom fatigue" after months of online coursework.Informal feedback from trainees suggests that their weariness with video conferencing was far outweighed by their eagerness for social interaction, and that new EYES relationships helped to combat pandemic-driven feelings of isolation.Accordingly, we will incorporate more opportunity for community-building, including more frequent social events and unstructured time before and after scheduled activities for mingling, both with and without EYES leadership present.

STUDY 2: RELATIVE IMPACT OF ONLINE ACTIVITIES VS. IN-PERSON EXPERIENCES
Beyond preserving trainees' access to the program, the EYES team was eager to assess the extent to which online activities supported target outcomes, specifically trainees' research skills and career development.In particular, we were curious to compare initial outcomes for trainees in the program's first three cohorts -who spent their first summer immersed in mentored, in-person research -to the outcomes of trainees in the 2020 cohort -who, by the end of their first summer in the program, had experienced exclusively non-research-based, online programming.The purpose of the comparison was to examine the impact of online enrichment activities, which generally require fewer resources and accommodate large numbers of participants, relative to mentored research experiences, which are generally resource-intensive and consequently, of limited availability.

Methods.
Analyses compared outcomes after one summer of programming between two groups.The In-Person group consisted of the 32 trainees in the 2017, 2018 and 2019 cohorts, who spent their first summer in the program immersed in mentored research.The Remote group consisted of the 13 trainees in the 2020 cohort, who participated in online skill and career development activities, but no mentored research, during their first summer in the program.Demographics are reported in Table 1, under Study 2. One additional member of the 2020 cohort was excluded from analyses because she participated in remote research her first summer, an experience incomparable to either group.
Participants completed a survey assessment via RED-Cap during their first week in the program ("pre" assessment; 100% response rate) and again eight weeks later, at the end of their first summer experience ("post" assessment; 97.8% response rate).Most survey measures were adapted from assessments of the Leadership Alliance Summer Research Early Identification Program (Ghee et al., 2016) and factors.Most common were conflicting responsibilities related to work, school, and family (61.9% of respondents), access to a suitable electronic device (28.6%), and access to a quiet, private space (28.6%).Nineteen percent of participants indicated emotional burden and 14.3% limited access to reliable internet.
In describing ways the EYES team helped to mitigate factors interfering with participation, 50.0% of trainees cited the team's understanding of individuals' circumstances (e.g., the program director "was so amazing and understanding when I informed her twice at the very last minute that I wouldn't be able to attend a session because I had to take my grandfather to the doctor").Other common themes included the fact that meetings were recorded (31.8% of coded responses), flexible program structure (e.g., "They allowed the students to have a choice of which activities they wanted to engage in depending on their situation"; 27.3%) and creating a safe, supportive environment (e.g., "They made an incredibly comfortable and safe program in which I never felt like less than anyone else there"; 18.2%).With regard to ways the EYES team could have mitigated factors interfering with participation, 82.6% of respondents had no suggestions, one trainee suggested a more concerted effort to schedule activities around trainees' availability, and three others provided suggestions for specific activities, including workshops on mental health and time management, and more frequent social events.
Collectively, trainees identified up to a dozen different activity options as best supporting their growth in a given domain.The faculty journal club series was most often cited for enhancing scientific skills and expertise (63.6% of participant responses); the workshop on networking for career exploration was most often cited for enhancing professional skills (22.7%); the workshop on navigating career pathways was most often cited for enhancing career knowledge and ambitions (40.9%); and finally, the family night was most often cited for enhancing trainees' sense of support and belonging (59.1%).
Discussion.Despite the fact that most trainees experienced obstacles to their participation in the 2020 EYES virtual summer session, most commonly with regard to competing responsibilities at school, work and home, all but three opted in to, and successfully completed, summer program requirements.In fact, the vast majority exceeded expectations with regard to their engagement in program activities.We conclude that our efforts to maximize flexibility through distinct program tracks, activity choice and make-up work options were successful in preserving program accessibility despite the compromising circumstances of the pandemic.We also conclude from trainees' strong engagement that the activities offered catered well to trainees' needs and interests.However, in response to trainees' feedback, we will incorporate the Training Early Achievers for Careers in Health Research program (Arora et al., 2011), comparable research-intensive, mentor-based programs to foster diversity in the biomedical workforce.
At pre and post assessments, trainees rated on a scale from 1 (none) to 5 (very high) their knowledge of 11 aspects of conducting research as a measure of research skill development.Items referenced broad concepts such as the overall research process, as well as technical aspects such as approaches to quantitative data analysis.The 11 aspects are listed in Figure 1a.
At pre and post assessments, trainees were also asked a series of career development questions to gauge their level of knowledge and interest in careers in biomedicine.In particular, they rated on a scale from 1 (strongly disagree) to 5 (strongly agree) the extent to which they agreed with the statement: "I have a solid understanding of the careers available to me in the field of biomedicine."Trainees also rated on a scale from 1 (definitely not interested) to 5 (definitely interested) their interest in pursuing a career as a scientific researcher.Additional items administered at post assessment only asked trainees to rate on a scale from 1 (strongly disagree) to 5 (strongly agree) their level of agreement with the statements "Overall, I am more confident about planning for a research career" and "My commitment to pursue a research career is stronger."Finally, they rated on a scale from 1 (not at all) to 5 (extremely) the usefulness of the program in increasing their readiness to engage in demanding research, improving their self-confidence as a researcher, and understanding more clearly what it takes to become a researcher.
Analyses were performed with SPSS 26 software.All survey items were treated as ordinal (Lovelace and Brickman, 2013).Frequencies, percentages, medians and ranges were calculated for each.The Mann-Whitney U test, a nonparametric alternative to the independent samples t-test, was used to evaluate between-group differences at either assessment.The test ranks all values for a given item from low to high and determines whether rankings between the two groups differ significantly.The test statistic, U, ranges from 0 to the product of the two sample sizes.The smaller the value, the more different the groups (Nachar, 2008).For items administered at both pre and post, the related-samples Wilcoxon signed rank test, a nonparametric alternative to the paired samples t-test, was used to evaluate within-group changes over the course of trainees' first summer in EYES.The test ranks the absolute values of the differences between pre and post ratings for a given item, then compares the sums of ranks for differences with negative versus positive signs.The closer the calculated Z statistic is to zero (on a scale from approximately -3 to 3), the more evenly the negative and positive differences are distributed, meaning less difference in ratings from pre to post.For all tests, p-values of 0.05 or lower were considered statistically significant.The full five-point range for each item was used for all analyses.However, to increase readability of the findings, Figure 1 illustrates just the percentage of respondents in each group who selected a rating of 4 or 5, indicating relatively high levels of perceived knowledge, agreement, interest, or usefulness, respectively.

Results.
Research Skill Development.The In-Person and Remote groups reported similar research knowledge at pre-assessment, with no significant between-group differences in mean ranks for any of the 11 aspects of conducting research (all U statistics > 117.5, all p-values > 0.08).Baseline levels were modest, with fewer than a third of trainees in either group rating their knowledge a 4 or 5 across items.There was one exception: half of the Remote trainees felt they were knowledgeable about ethical aspects of research (Figure 1a).The groups did exhibit differential gains in research knowledge over the course of the summer.Wilcoxon signed rank tests indicated that post-assessment ranks were significantly higher than pre-assessment ranks for all 11 aspects within the In-Person group (all Zs < -3.13, all p-values < .01).Comparatively, within the Remote group, post-assessment ranks were significantly higher than pre-assessment ranks for just 3 items: professional aspects of research (Z = -2.09,p = 0.04), literature and basic concepts in a specific research area (Z = -2.16,p = 0.03) and critical reading skills and tools for analysis of text (Z = -2.65;p = 0.01).
Career Development.At pre-assessment, 53.3% of the In-Person group and 45.5% of the Remote group reported that they had a solid understanding of careers available in biomedicine (Figure 1b).The mean ranks of the groups did not differ significantly (U = 158.0,p = 0.52).Wilcoxon signed rank tests indicated a significantly greater understanding of available careers in biomedicine at post assessment for both the In-Person (Z = -2.92,p < .01)and Remote (Z = -2.36,p = 0.02) groups.Consistent with the self-selective nature of the program, 86.7% of the In-Person group and 91.7% of the Remote group indicated a high degree of interest in a research career (Figure 1c).The mean ranks of the groups did not differ significantly (U = 127.5,p = 0.11).We observed a decline in both groups by post assessment, with 70.0% of In-Person and 75.0% of Remote reporting a high degree of interest.However, these changes were not significant (In-Person: Z = -1.27,p = 0.20; Remote: Z = -1.52,p = 0.13).
Mann-Whitney U tests revealed no significant between-group differences on any of the cross-sectional, post assessment measures of career development, specifically trainees' confidence in planning for a research career (U = 149.5,p = 0.55); their commitment to pursuing a research career (U = 129.5, p = 0.25) (Figure 1d); or their assessment of the program's usefulness in enhancing their understand-  ing of what it takes to become a researcher (U = 113.5,p = 0.10); their readiness to engage in demanding research (U = 161.0,p = 0.80); and their self-confidence as a researcher (U = 145.0,p = 0.48) (Figure 1e).

DISCUSSION
With regard to research skills knowledge, trainees who participated in an intensive research experience their first summer unsurprisingly had a clear advantage, reporting gains for every aspect examined.Trainees who participated exclusively in online enrichment activities reported gains for three of the more general aspects of research knowledge, including professional aspects of research, literature and basic concepts in a specific research area and critical reading skills and tools for analysis of text.These findings underscore the unique value of mentored research experience for the development of specialized research skills and expertise.Accordingly, the EYES 2021 summer session includes full-time research experiences for all trainees.These include fully-remote options for high school students, who are still not permitted on campus as part of the university's COVID-19 protocol.
With regard to career development, the In-Person and Remote groups were remarkably similar.Both felt more knowledgeable about careers in biomedicine, and research careers in particular.They felt more confident in planning for a research career and in their ability to do the work.Finally, the majority of trainees in both groups retained a high degree of interest in, and strengthened their commitment to, a research career.Career-specific knowledge and attitudes are key to forming aligned ambitions, or congruence between one's perceptions of a particular career and the actions needed to attain it, and reality.Youth with aligned ambitions demonstrate more realistic and systematic approaches to pursuing career goals and successful career entry (Schneider, 2009).Yet, the experiences and mentorship that facilitate aligned ambitions can be difficult to access, particularly among underrepresented minorities.Although these findings are preliminary (data are from a single program cycle with a small sample size), they encourage us to consider the potential of online enrichment activities as a relatively low-cost, high-capacity approach to foster career knowledge, self-confidence, and commitment among young people considering a future in cancer research.

CONCLUSION
Redesign of the EYES program was born of necessity.In the chaotic first months of the COVID-19 pandemic, our virtual activities succeeded in addressing the needs of trainees eager for intellectual, social and financial support.Now, as we plan for a post-COVID-19 future, the success of our initial efforts has inspired us to pursue virtual activities as a permanent component of research training and education outreach initiatives.For program leaders making similar considerations, we offer three takeaways from our experience to date.
First, the switch from an in-person to virtual format need not compromise activity quality or effectiveness, provided program leaders are sensitive to the limitations of the virtual setting.In designing our virtual program, we prioritized those activities we expected to translate naturally to Zoom, such as the faculty journal club, the book club series, skill-building workshops and panel discussions.Social activities and community outreach events, which derive much of their value through casual and simultaneous interactions between pairs or small groups of individuals, felt lacking in a virtual meeting room.
Second, the shift to the virtual realm has been both widespread and indefinite, creating rich new opportunities for trainees to engage with peers, mentors and role models across the professional science world.The AACR conference described above is one example.In summer 2021, our trainees will have unprecedented access to the university's tumor boards, which are now hosted via Zoom.We have also arranged joint activities with our YES colleagues at the University of Pittsburgh and the University of Kentucky, and we will welcome guest speakers and workshop facilitators from across the country, exposing trainees to a broader and more diverse network than could be accomplished through our institution alone.
Third, the incorporation of virtual elements is a promising means to increase program accessibility.For example, we plan to make virtual, synchronous activities the norm for our monthly academic year programming, greatly improving access for those attending colleges outside of Chicago.We also plan to leverage virtual programming to support the career ambitions of the 96% of EYES applicants not admitted to the program, due almost entirely to limited capacity of research placements.This summer, we opened our weekly faculty lecture series and a selection of career development workshops to those on our mailing list, who have not yet been accepted to the program.As of the time of this writing, more than 250 registered to participate.
In these ways, the unexpected shift to a virtual format promises to have a long-lasting, positive influence on our broader cancer education strategy.

ASSOCIATED CONTENT
Supplemental material mentioned in this manuscript can be found uploaded to the same webpage as the manuscript.

Figure 1 .
Figure 1.Relative outcomes after one summer of programming for the In-Person group (trainees in the 2017, 2018 and 2019 cohorts, who spent their first summer in the program immersed in mentored research) and the Remote group (trainees in the 2020 cohort, who participated in online skill and career development activities, but no mentored research).An asterisk indicates a p-value of .01 or less.

Table 2 .
Virtual skill-building and career development activity options for trainees in summer 2020, by theme.