Comparing Online vs. In-Person Outcomes of a Hands-On, Lab-Based, Teacher Professional Development Program: Research Experiences for Teachers in the Time of COVID-19

: In 2012, engineering was adopted as a content area in K-12 science education. Yet few science teachers have engineering content or pedagogical content knowledge. Teachers learn how to teach engineering through professional development experiences, for which Darling-Hammond et al. (2017) outline seven characteristics of effectiveness. This mixed-methods study compared three cohorts of the NSF-funded Research Experiences for Teachers (RET) participants at one university. Two cohorts of in-service teachers experienced RET on-site, and one (summer 2020) experienced the program 100% remotely, due to COVID-19. Research questions explore 1) the extent to which the RET program delivered in-person and remotely reflected the seven characteristics of effective PD; and 2) whether program outcomes of the hands-on, lab-based program could be achieved remotely. This study found that six of the seven characteristics of effective PD were fully evidenced in all three RET cohorts, and one was partially evidenced. Program outcomes were achieved at high and comparable levels for all cohorts, though the nature of the experience varied by mode of delivery. This research contributes to existing literature that finds no inherent differences in the quality of teacher professional development delivered in-person or remotely.


INTRODUCTION
Hands-on, laboratory experiences are considered essential in STEM education. When escalation of the COVID-19 pandemic in March 2020 locked Principal Investigators (PIs), lab staff, and students out of college and university labs, educators worldwide had to strategize alternatives (Chen et al., 2020). Also affected were K-12 teachers who benefit from intensive, professional learning experiences that take place on college and university campuses during the summer. In March 2020, leaders of the Stanford Research Experiences for Teachers (RET) programs for K-12 teachers were finalizing participant selection, recruiting staff, and preparing program materials for summer 2020 when University restrictions due to COVID-19 closed all labs. It was unknown at that time whether the labs would open in time for the summer program, which was scheduled to begin in June.
The National Science Foundations' (NSF) RET solicitations explicitly state that experiences need to be on-site at the college or university (NSF, 2019). For summer 2020, due to extraordinary circumstances, NSF permitted program leaders to design and deliver programs in accordance with the policies of the governing institution. RET program directors had the option to suspend their program, deliver their program remotely, or run the program using some combination of remote and/or COVID-19-safe in-person sessions. Program leaders faced an unprecedented challenge shared by all STEM educators: Can a hands-on, lab-based, professional development program successfully achieve its program outcomes if it is delivered partially or completely remotely? After extensive deliberation and consultation with lab faculty, Stanford program leaders decided to pilot a 100% virtually delivered summer program to a limited number of teachers. The mixed-methods, evaluation study reported here relies upon Darling-Hammond et al.'s 2017 research into high quality, teacher professional development to explore the extent to which the RET program met the seven qualities of effective teacher professional learning opportunities and whether a lab-based teacher professional learning experience can be effectively implemented through an online format. sional learning that results in changes to teacher knowledge and practices, and improvements in student learning outcomes" (p.2). They scoured research literature looking for research-based teacher professional learning programs that resulted in changed teacher practices and improved student outcomes. To be included, programs had to have an experimental design or comparison group and analyze student outcomes, controlling for context and student variables. Their search identified 35 such programs, results of which were carefully re-analyzed. The 2017 report summarizes seven characteristics of effective teacher professional learning that were found to be common across the 35 studies: 1. Is content focused 2. Incorporates active learning utilizing adult learning theory 3. Supports collaboration, typically in job-embedded contexts 4. Uses models and modeling of effective practice 5. Provides coaching and expert support 6. Offers opportunities for feedback and reflection 7. Is of sustained duration (Darling-Hammond et al., 2017, p.4)

The NSF's Research Experiences for Teachers (RET)
Program as a Teacher Learning Program. The RET program in engineering and computer science was initiated by the NSF in 2001 (Russell and Hancock, 2007). K-14 educators are placed in college or university labs with research faculty in order to develop hands-on experiences conducting authentic science and engineering research. The program intends that teachers use their RET experiences to bring science concepts alive in their own classrooms in order to inspire students to pursue STEM careers. Key features of NSF's RET program include: • Active collaborative partnerships between K-12 and/or community-college educators, and college and university faculty and students • Authentic summer research experiences • A common cohort experience among participating educators • Integration of summer research experiences into educators' classrooms (NSF Program Solicitation https://www. nsf.gov/funding/pgm_summ.jsp?pims_id=505170).
These features align with several of the characteristics of effective teacher learning summarized by Darling-Hammond et al. (2017), cited above. Although participants nationwide rate RET experiences highly, research has shown mixed results in actually changing teaching practices and improving student outcomes. Miranda and Damico (2015) analyzed data from 14 science teachers who participated in a yearlong RET program designed to promote inquirybased instruction. Teachers evidenced significantly higher numbers of teacher-student interactions and stronger teacherstudent relationships, and a significant overall increase on Reformed Teaching Observation Protocol (RTOP) scores. Nevertheless, no statistical differences were observed in teachers' lesson design and implementation, propositional knowledge, or procedural knowledge. In their study of nine teachers in a three-week, intensive STEM pedagogy workshop, Singer et al. (2016) found that the extent of social interaction during teacher professional development was the only variable that significantly influenced science teachers' classroom practices and efficacy beliefs. Through structural equation modeling (SEM), Southerland et al. (2016) found that changes in RET teachers' practices resulted directly from teachers conducting research focused on improving their understanding of science. Social interaction within the research experience was an indirect factor in changing teaching beliefs, which in turn led to enhanced teaching practices. Lichtenstein et al. (2016) reported that high school students found engaging pedagogical strategies promoted in their teachers' RET program were significantly more motivating than conventional teaching strategies, but student outcomes were not assessed. Ragusa and Mataric (2016) found improved teacher performance and self-efficacy (29% and 24%, respectively), as well as significant student gains in science knowledge, literacy, and science interest and motivation as a direct result of their teachers' participation in a RET program.

Online Teacher Professional Development (OTPD).
Experimental and comparison group studies of online teacher professional development (OTPD) vs. in-person professional development (PD)-based on similar contexts and duration, as well as post-observation of teaching behaviors and/or student outcomes-show no patterns favoring one or another modality (Fisher et al., 2010;Fishman et al., 2013;Masters et al., 2010, Powell et al., 2010. In one randomized, experimental study of high school biology teachers, OTPD teachers scored significantly higher than teachers in the in-person PD condition on content knowledge, pedagogical knowledge, and the use of digital media; however, there were no significant differences in student outcome scores. Furthermore, attrition from the OTPD group was notably higher than that from the in-person group (chi-square=3.078, df=2, p=.07; Goldenberg et al., 2014). Binmohsen and Abrahams (2020) found that teachers who experienced OTPD professional development scored significantly higher on a classroom observation instrument than teachers who experienced the same training in-person. Although comparisons of OTPD with PD are few, mixed results suggest that professional development outcomes are not inherently dependent on the modality of training.
Research Questions. Taken together, studies of OTPD that include empirical evidence of improved teacher behaviors or student outcomes show mixed results-just as do studies of in-person teacher professional learning (Bissonnette and Caprino, 2015;Hill, 2009;Lee et al., 2020;McChesney and Aldridge 2018). No evidence confirms that one modality is inherently more effective than another. In light of that, it is reasonable to assume that the characteristics identified in successful teacher professional learning programs that alter teacher behaviors and student outcomes (i.e., Darling-Hammond et al., 2017) would apply to online as well as in-person teacher professional learning.
This study analyzes Stanford's eight-week, summer RET program, originally designed as an in-person, hands-on, engineering laboratory experience. When COVID-19 made access to campus labs impossible, program leaders switched to 100% virtual program delivery. Program leaders as well as faculty questioned whether the benefits of a lab-based professional learning experience could be realized in an online format. Our research questions are: 2. To what extent do immediate outcomes of 100% virtual program delivery compare to outcomes of in-person program delivery?

METHODS
Program Description. The Stanford RET program is a partnership between Stanford University's Office of Science Outreach (OSO) and Ignited. Ignited has been providing paid, industry-and university-based teacher professional learning experiences throughout California's Silicon Valley for 35 years, reaching more than 2,000 teachers. The grant term for the Stanford RET program was three years, involving three teacher-fellow cohorts: 2018 (Cohort 1, C1), 2019 (Cohort 2, C2), and 2020 (Cohort 3, C3). Program elements were the same each year: 1) Laboratory Research (~35 hours/week), 2) STEM content knowledge and pedagogy (4 hours/week), 3) individualized sessions with an instructional coach (minimum 30 minutes/week), and 4) development of Education Transfer Plans (ETPs ~5 hours/week).
During the program, RET teacher-fellows conducted laboratory research, guided by research mentors, who were PIs or advanced graduate students in each lab. All three teacher-fellow cohorts participated in science and engineering labs for eight hours each day Tuesdays-Fridays. In 2018 (C1) and 2019 (C2), research occurred in the physical laboratories on the University's campus. In summer 2020 (C3), program leaders, teacher-fellows, research mentors, graduate students, and the instructional coach all worked remotely. Each of the three years, weekly professional learning-specific sessions occurred on Monday mornings. In 2018 and 2019, the face-to-face sessions ran for four hours, but in 2020, the program leaders, seeking to avoid Zoom-fatigue, reduced the morning sessions to two hours. Instead of including certain exercises during training, teacher-fellows arrived to the Monday sessions having completed the assigned preparatory work, which in past years had been incorporated into the four-hour sessions.
In all years, guest speakers presented on a range of relevant topics for 30-60 minutes (e.g. design thinking, virtual reality for science education, heat transfer at the nanoscale, the factors that influence students' decisions to pursue STEM, and environmental education), followed by discussion. Aspects of engineering, computer science, and pedagogy were integrated across all of the weekly sessions. During 2019 and 2020, Design Thinking (DT) was integrated across Monday PD sessions. In 2018, however, the expert scheduled to present the DT component was unable to participate and a replacement could not be found in time. Therefore, the comparisons of DT do not include C1, the 2018 cohort. The Monday sessions usually culminated in a debrief of teacher-fellows' situations with their research mentors and preparing for the next session.
Teacher-fellows in C1-C3 met weekly in "families" with the instructional coach (a currently practicing classroom teacher and former Ignited teacher-fellow) and Ignited teacher-fellows hosted by other organizations. In addition, the instructional coach provided individual support for curriculum development, advising teacher-fellows on how to translate their research skills and experiences into lessons for their students. Teacher-fellows also created and maintained their own online Reflection Journals. These activities took place for all cohorts; only C3 (2020 teacher-fellows) participated remotely.
Sample. This study focused on three cohorts of RET teacher-fellows who participated in the eight-week summer program in 2018, 2019, 2020. Table 1 summarizes the demographic information of all three cohorts. Teachers applied for the summer fellowships through Ignited (http:// www.igniteducation.org), which places K-14 teachers in internship experiences in tech companies and research labs throughout California's Silicon Valley. In order to be eligible, teachers must have completed two years' full-time teaching in K-14 contexts, commit to working the entire eight weeks of the summer program, and demonstrate a commitment to teaching for at least three years after the program. Applicants are coached during the application process by Ignited staff, but hiring decisions are made by industry or lab supervisors ("mentors"). Teacher-fellows may participate in fellowships for more than one summer.
Applications for fellowships from teachers whose background and interests align with projects in laboratories at Stanford are forwarded to Stanford RET staff, who then share with faculty willing to host teachers during the summer. The 2020 cohort differed in some respects from prior cohorts. C3 was limited to eight participants, since this was the number that faculty could accommodate who were willing to pilot 100% virtual lab experiences. In interviews, lab PIs indicated that their decision whether to host a C3 teacherfellow in under COVID-19 restrictions was based on 1) PI's previous experience working with RET teacher-fellows, 2) a teacher-fellow's previous experience participating in a RET program, and 3) PI's having a project available that a teacher-fellow could engage in remotely. Four of the eight PIs who were willing to experiment with remote delivery had hosted teacher-fellows at least once previously.
Of the eight C3 teacher-fellows, four had participated in RET previously, and a fifth had participated in a similar summer intern experience in a corporate setting. By faculty request, the four returning C3 teacher-fellows were placed in labs in which they had worked previously. C3 teacher-fellows had more years' teaching experience than did teachers in C1 rating the virtual experience. Participants who had attended prior years were asked to compare the virtual to the in-person experience. Response rates on all pre-and post-surveys for all cohorts was 100%.
Interviews. The evaluation team conducted semi-structured interviews with all eight teacher-fellows in C3 and six teacher fellows in C1 and C2 (Drever, 1995;Posner, 1982).
Interviews of all participants throughout all three years took place by Zoom. Informal interviews were conducted with all C3 participants during weekly or bi-weekly check-ins. Over the course of the eight-week session, focal participants were interviewed informally as opportunities arose, and formally at least twice. In some cases, participants were interviewed a third time, if evaluation questions remained unanswered. In order to understand the C3 fellows' virtual experience, two evaluation team members divided participants between them and observed lab sessions, Monday professional learning sessions, and some independent work periods. During C3, the evaluation team also formally interviewed the research mentors who hosted a teacher-fellow in each lab. These interviews focused on the experience of having the teacher-fellow as part of the lab team, the teacher-fellow's contributions to the lab, and the value to the research mentor of working with teacher-fellows. Also during C3, the Ignited instructional coach was formally interviewed twice and informally interviewed throughout the eight-week session. Interview questions focused on the progress of teacherfellows' Education Transfer Plans (ETPs)-the lesson plans that translate teachers' lab experiences into classroom lessons.
Observations of Labs, Meetings, and Events. Evaluators attended teacher-fellow orientation sessions for all three cohorts. During C3, they also attended kick-off meetings for research mentors. In addition, project evaluators provided formative feedback to program leaders during weekly meetings and occasionally through emails. Each year, teacher-fellows shared their final projects during an end-program symposium with their cohort members. During C3, teaching colleagues, administrators, lab colleagues, their own students, and whomever else they chose to invite attended the virtual symposium. The evaluation team observed all of these workshops and the symposium.
The Researchers. Data were collected as part of a threeyear, evaluation study of the Stanford Research Experiences for Teachers program, conducted by Quality Evaluation Designs (QED), a for-profit research and evaluation firm that has conducted research for K-12 school districts, colleges and universities, foundations, and government entities nationwide for twenty-five years. Data were collected by two seasoned evaluators, each with over 25 years' experience in and fewer years' teaching experience than Cohort C2. C1 teacher-fellows compared to C2 had a higher proportion of teachers in their first through fourth years; however, the two cohorts had similar proportions of teachers with 11 or more years' experience. Out of the eight teacher-fellows in 2020 (C3), three taught high school, three taught middle school, one taught elementary school, and one taught community college. Teacher-fellows from all three cohorts taught either science (5), math (1), or engineering/computer science (2). All of the teacher-fellows came from either urban and/or suburban (no rural) areas and the breakdown by percentage in each category was similar across cohorts. Overall, C3 served fewer schools with Free and Reduced Lunch programs than had C1 (2018) and C2 (2019).

Mixed-Methods Research.
This was a mixed methods study based on program evaluation data collected in a convergent, mixed-methods design (Breitmayer et al., 1993;Fetters et al., 2013). Data included surveys, interviews, and observation notes. Survey changes after 2018 limited comparisons between all three cohorts, so most data compare C2 and C3 outcomes. Data collection was expanded in 2020 in order to assess the effects of online program delivery on teacher-fellows as well as PIs and science mentors. Means on survey items were summarized using conventional descriptive statistics. Qualitative data (surveys and interviews) aligned with narrative methodology (Connelly and Clandenin, 2006;Lichtenstein, 1997). As described by Connelly and Clandinin, narrative methodology "entails a view of the phenomenon. To use narrative inquiry methodology is to adopt a particular view of experience as the phenomenon under study" (Connelly and Clandinin, 2006, p. 479).
Integrity and authenticity of data collected were assessed based on patterns of participants' responses within and across cohorts, convergence of interview and survey data over time and within years, continuity between researcher-observed experiences and participants' meaning-making over time, and convergence of meaning-making across program contexts. We initially used Miles and Huberman's (1994) and Miles et al.'s (2013) cluster coding techniques for analysis of the semi-structured interviews and open-ended survey items.
The process of open coding (particularly of interviews) revealed themes convergent with both quantitative and qualitative survey data (Saldaña, 2015).
Pre-/Post Surveys. Teacher-fellows in all three cohorts completed pre-and post-surveys that collected demographic information, background/experience with design thinking, and ratings of RET program effectiveness. C3 surveys included questions regarding how included teacher-fellows felt in the labs to which they were assigned, how connected they felt to their fellow cohort members, as well as items science teaching and evaluation methods. The lead author is QED founder and principal. All data were collected according to Ethical and Independent Review Services IRB Protocol #18108-01. Quantitative data were analyzed using Excel with Qi Macros.

RESULTS AND FINDINGS
Data were analyzed according to the Darling-Hammond et al. (2017) framework for effective professional learning. These features of teacher professional learning are found in programs that change teacher behaviors and improve student outcomes. The following sections provide evidence of the effectiveness of the RET program in meeting each criterion.

Feature 1. Professional Learning is Content Focused.
[Professional learning] that focuses on teaching strategies associated with specific curriculum content supports teacher learning within teachers' classroom contexts. This element includes an intentional focus on discipline-specific curriculum development and pedagogies in areas such as mathematics, science, or literacy (Darling-Hammond et al, 2017, p. v).
The Stanford RET program focused on developing interdisciplinary content across STEM in alignment with the Science and Engineering Practices (SEPs) of the Next Generation Science Standards (NGSS). Discipline-specific curriculum development was based on: 1) teachers' laboratory research and 2) Monday professional learning sessions focused on STEM content and pedagogy.
Each Monday STEM professional learning session included a guest speaker who presented a discipline-specific presentation about their current research (e.g. heat transfer, human impacts on the environment, etc.). The Monday sessions augmented the teacher-fellows' experiences in the laboratory settings with a range of science and engineering content. Exit surveys asked C2 and C3 teachers the likelihood of their integrating examples from those presentations into their teaching. In both cohorts, 88% of teacher-fellows reported they would very likely integrate ideas shared by research faculty into their teaching.
As with many RET programs nationwide, content-focus through hands-on laboratory experiences is a predominant feature of the Stanford summer experience. Post-survey items asked teacher-fellows to reflect on the influence of the RET program on their content and disciplinary knowledge. Ratings across cohorts were consistently high. Table  3 shows that teacher-fellows' mean ratings for exposure to content-specific research and disciplinary understanding were comparable across all three cohorts.
Teacher-fellows' comments across cohorts emphasize the value of the RET experience for enhancing their content and disciplinary knowledge: In 2020 (C3), teacher-fellows partnered with research mentors conducting research for labs in neurotechnology, assistive robotics, marine science, mechanical engineering, civil and environmental engineering, materials science, and a biomimetics and dexterous manipulation lab. Two examples of lab work that C3 teacher-fellows engaged in for mechanical engineering PIs who had participated in RET previously include: 1) a teacher-fellow helped design, conduct, and analyze test data from gripping experiments with adhesives and spines, using CAD, and 2) a teacher-fellow developed curricular modules to introduce machine learning and its engineering applications to high school students. Two examples of lab work that C3 teacher-fellows engaged in for PIs who had not participated in RET previously include: 1) in an assistive robotics laboratory, a teacher-fellow helped develop the algorithm for and programmed (using Python and Java) an app that assists in communicating using American Sign Language, and 2) in a materials science lab, a teacher-fellow assisted in designing and implementing new materials and instruments for advanced brain-machine interfaces. C3 teacher-fellows were able to conduct science research remotely and described a wide range of projects. In an interview, one teacher-fellow said:  RET teacher-fellows' Educational Transfer Plans (ETPs) serve as the vehicle by which fellows transfer their summer experience into their instructional practice. Table 3 shows how ETPs from the RET 2020 cohort reflect the discipline-specific content knowledge. Active learning engages teachers directly in designing and trying out teaching strategies, providing them an opportunity to engage in the same style of learning they are designing for their students. Such PD uses authentic artifacts, interactive activities, and other strategies to provide deeply embedded, highly contextualized professional learning. This approach moves away from traditional learning models and environments that are lecture based and have no direct connection to teachers' classrooms and students (Darling-Hammond et al, 2017, p. v).
Engaging laboratory experiences naturally involve active learning. Typically, RET lab experiences are highly rated because they are hands-on, and because teachers are excited working in labs alongside world-renowned scientists and engineers. An important question for RET leaders in 2020 was whether the benefits of laboratory placements could be realized in a 100% remote learning environment.
Teachers spent approximately 35 hours each week engaged in lab projects. This was as true for the 2020 virtual cohort as it was for cohorts from prior years. All cohorts experienced hands-on learning, yet the nature of the "handson" varied. In C1 and C2, when teacher-fellows were in labs, they often used expensive, highly sophisticated equipment to conduct experiments and collect data. They also had handson experiences analyzing data and conducting auxiliary research that added value to lab findings. C3 teacher-fellows, who did not have hands-on experiences with lab equipment, had more experiences analyzing data and conducting auxiliary research. When reflecting on hands-on learning, teacher-fellows from C2 were more likely to reference their in-lab experiences: Distance learning module for middle-schoolers that enables students to make the connection between the health of the ocean and the health of humans. ETP focuses on an environmental justice storyline that recognize inequities in the health and environmental effects to humans of degraded marine environments.

C3.4 Neuroscience Department:
Researches advanced brainmachine interfaces that allow computers to communicate with the brain directly.
A laboratory experience that will help elementary students better understand the neurobiology of the eye, then demonstrate their understanding through a design challenge to build a camera obscura. Students learn how current technologies involve similar concepts and phenomena in order to excite students about potential jobs in the field.

C3.5 Civil and Environmental
Engineering: Creating an analyzing wind measurement data.
A curriculum unit to create physics virtual labs for high school students based on actual data and authentic science research.

ETP challenges Community
College students to design, conduct and analyze test data from gripping experiments with adhesives and spines. Students will determine whether gripping systems perform as well in space as they do in earthbound labs.

Table 3. Cohort 3 (2020) Labs and Education Transfer Programs (ETPs)
It was evident through interviews and post surveys that C3 teacher-fellows felt that they had experienced hands-on learning. C3 fellows were more likely to reference auxiliary research they conducted or their ETP's, which translated their remote-lab experiences into lesson content, often in formats for online delivery: The Stanford RET program grant was built around Design Thinking (DT), as promoted by the Hasso Plattner Institute of Design (Stanford d.school), described by Sheppard and Goldman (2017). DT was presented as a way to strategize project-based learning (PBL). As a result, PBL was promoted in the labs (where teacher-fellows experienced it) and in Monday STEM professional learning sessions, where teacher-fellows focused on translating their lab experiences into classroom lessons.
Teacher-fellows discussed, reflected upon, and shared their ETPs in Monday STEM professional learning sessions, with their research mentors in labs, and with their instructional coach. Teachers in all cohorts reported experiencing the value of hands-on, inquiry-based learning and, as evidenced below, most were moved to include those elements in their own classrooms, as reflected in this teacher's comment:

Feature 3. Supports Collaboration, Typically in Job-Embedded Contexts.
High-quality PD creates space for teachers to share ideas and collaborate in their learning, often in job-embedded contexts. By working collaboratively, teachers can create communities that positively change the culture and instruction of their entire grade level, department, school and/or district (Darling-Hammond et al., 2017, p. v).
The RET program consistently promoted collaboration and community among teacher-fellows, program leaders, research mentors, and lab personnel. By all accounts, the C3 (2020) teacher-fellows felt connected to their labs and lab personnel, even though contact was virtual. In one case, the research mentor and teacher-fellow texted each other multiple times each day to problem solve. Another research mentor was sure to add the teacher-fellow to all mailing lists for the lab so the fellow could access the same seminars that lab personnel would attend. This research mentor met with the teacher-fellow to set goals twice each week and they talked every day:

I think [the teaching fellow] is getting as close to the lab experience as she can get. It's important that she can text me at any time. I also got her on the mailing lists because she should be included in all the seminars that we would have gone to. (C3 Research
Mentor) Figure 1 shows the frequency with which teacher-fellows reported interacting with different categories of lab personnel. N/A refers to laboratory research experiences that did not involve that category of personnel or where the PI of the lab was the same person as the research mentor. Figure 1 shows that six of eight C3 teacher-fellows interacted several times each week with the lab PI and research mentors. None of the eight C3 teacher-fellows reported communicating fewer than three times/week with lab staff, and most reported considerably more contact. In fact, the constraints of COVID-19 provided an unexpected benefit. Because professional conferences were cancelled or held virtually, faculty were not traveling, and so they were more available to meet with teacher-fellows than in previous years. Only one of the eight teacher-fellows interacted with the PI just once each week. One teacher-fellow described her regular contact: I met daily with someone from the lab, and attended Friday meetings to start. I also had a one-on-one with the head of the lab each week. He offered to answer any questions I had...Everyone was incredibly welcoming and touched base with me more as the summer went on. (C3 Teacher-Fellow) C3 teacher-fellows were asked what the lab personnel did that made them feel connected to the labs. Responses included: • Inclusion in lab and department meetings • Lab members initiating meetings with teacher-fellows • Lab members providing feedback on teacher-fellows

Education Transfer Plans (ETPs)
• Conversations about topics other than the lab research-personal connections • Lab members alerting teacher-fellows about materials relevant to what the teacher-fellow was working on, and/or K-12 related articles or resources • Sharing online conferencing tools • Alerting teacher-fellows to science and engineering opportunities Teacher-fellows also benefited from collaboration with members of their cohort. In both C2 (in-person) and C3 (100% virtual) program delivery, 100% of teacher-fellows reported in the post-survey that they would very likely use ideas and resources they got from other cohort members.
Working remotely might have emphasized to C3 program participants the collaborative skills necessary for conducting scientific research. One teacher-fellow described how meaningful it was to see how the lab members interacted with one another to share information, question one another, take on various tasks, and present their findings. The teacher-fellow was inspired by how lab partners worked together to develop tools for patients suffering from Alzheimer's and Parkinson's diseases. She felt that she would be able to use her virtual experience in the lab to provide concrete examples for her students of the importance of working together in teams and recognizing the diverse skills that every member of a team brings to a project.  The design and implementation of Stanford's RET program is similar to other RET programs nationwide in integrating models of adult learning through hands-on lab experiences, interactive teacher professional learning sessions, and project-based deliverables (i.e., ETPs). As mentioned earlier, the entire design of this RET program was based on active learning, including modeling effective practices. As teacher-fellows are integrated into labs, they are tasked with using meaningful, disciplinary content knowledge to solve authentic science and engineering problems. The design of the RET program encouraged teacher-fellows to translate those experiences into classroom practice. This translation process was promoted in Monday STEM professional learning sessions that both taught and modeled active learning, as well as in lab meetings and in weekly sessions with the instructional coach, who supported teacher-fellows in creating PBL-oriented ETPs.

Curricular models and modeling of instruction provide teachers with a clear vision of what best practices look like. Teachers may view models that in
As noted above, the thrust of the RET program was to teach design thinking (DT), as practiced by the Stanford d. school, in order to promote PBL in teacher-fellows' classrooms. The program's instructional coach was instrumental in working one-on-one with teacher-fellows to develop PBL-oriented ETPs. DT, a Stanford d.school method for designing PBL, was taught and practiced during the Monday STEM PD sessions. Research activities, STEM sessions, and coaching incorporated into the program provided models to guide teacher-fellows' implementation of PBL in their classrooms.
Using a retrospective pre/post survey design, the evaluation team asked teacher-fellows in C2 and C3 to consider the frequency of implementing PBL in their classrooms before the summer program, and the likely frequency after. Figure 2 shows substantial pre/post gains in both 2019 and 2020. The 2019 cohort shifted from 20% implementing PBL 3 or more times/term before the program to 56% intending to implement PBL 3 or more times after the program. The 2020 (virtual) cohort shifted from 37.5% implementing PBL 3 or more times/term before the program to 75% intending to implement PBL 3 or more times after the program. There is no significant difference in gains between 2019 vs. 2020 cohorts (Fisher's Exact Test,df=1,N=28;p=1.0).
Teachers in C2 and C3 commented on how experiencing active learning in the labs and Monday STEM professional learning sessions influenced their understanding of and confidence to implement active learning in their classrooms: The Design Thinking workshops inspired me to do more activities with students that incorporate empathy. (C2 Teacher-Fellow) I really liked all of the little activities we did that help your brain to start thinking differently in order to do design thinking better. (C2 Teacher-Fellow)

I feel like I picked up a new tool [Design Thinking] to drive teaching methodology. (C3 Teacher-Fellow)
As teacher-fellows participated in lab activities, they were professionally inclined to consider how to integrate what they were learning into their own classrooms. Ongoing development of their ETPs encouraged these connections. We asked them the extent to which their summer experiences are likely to influence their use of hands-on learning in their classrooms. Responses from C3 teacher-fellows, who experienced the program 100% remotely, were especially revealing: A LOT. I think it will be really important to have unique PBL summative assignments as we are teaching online. (C3 Teacher-Fellow)

This summer experience encouraged me to incorporate more technology projects into my classroom instruction. (C3 Teacher-Fellow)
I will try it more often than just the final team project for the semester. (C3 Teacher-Fellow)

I really enjoyed [another teacher-fellow's] presentation on how to get kids into the Design Thinking mindset. I think with an intro like that, it is easier to get the kids thinking outside of the box. (C3 Teacher-Fellow)
C3 participants also benefited from a form of hands-on learning that was not experienced by in-person cohorts; namely, gaining facility navigating online collaboration. Comments include:  (Darling-Hammond et al., 2017, p. vi).

Coaching and expert support involve the sharing of expertise about content and evidence-based practices, focused directly on teachers' individual needs
Above, we noted the frequency of contact between lab staff and teacher-fellows; in particular, frequent interactions with PIs and research mentors. In addition, each teacher-fellow in the Stanford RET program is part of a "family" of teacher-fellows supported by an instructional coach who shares their pedagogical content knowledge and expertise, helps teacher-fellows translate their STEM research experience into their classroom instruction (through the ETP), and provides one-on-one individualized support. The program requires that the instructional coach meet with each teacher-fellow for a minimum of 30 minutes/week. Teacher-fellows can and usually do request more time. C3 teacher-fellows reported spending an average summer total of 7.5 hours with their instructional coach, compared to 4.8 hours for C2 teacher-fellows. This is especially notable because typical meetings with the instructional coach during C1 and C2 took place on campus and lasted about an hour, which often included a walk around the grounds. However, C3 meetings happened virtually, and were therefore more focused, typically lasting 30 minutes. The fact that C3 teacher-fellows reported spending more time with the instructional coach than C2 teacher-fellows suggests that they received significant coaching and expert support despite, as well as because of, experiencing the program remotely.
C3 teachers sought more meetings with the instructional coach compared to prior years in part due to the remote delivery of the program. Because C3 teachers were not physically in labs, there were gaps between when they completed one lab task and were assigned a new one. During these gaps, teachers developed their ETPs. At least one teacher created three ETPs instead of the one required one. In both C2 and C3, 100% of teacher-fellows reported that the amount of time they spent working with their instructional coach was neither too little nor too much, but was just right. Two teachers from C3 described the willingness of program personnel to support their learning during the program:  Teacher-fellows in C2 and C3 rated each Monday STEM professional learning sessions (speakers, tours, other activities) on a scale from 1 to 4, with 1 indicating it was no or low value and 4 meaning it was extremely valuable. The overall mean ratings were 3.4 for C2 (2019) and 3.7 for C3 (2020). Differences are not statistically significant: X 2 (df=1, N=298)=2.53; p=.11).

Feature 6. Offers Opportunities for Feedback and Reflection.
High-quality professional learning frequently provides built-in time for teachers to think about, receive input on, and make changes to their practice by facilitating reflection and soliciting feedback. Feedback and reflection both help teachers to thoughtfully move toward the expert visions of practice (Darling-Hammond et al., 2017, p. vi).
Although this feature of effective professional learning was integrated into the Stanford RET experience, we have less evidence of implementation than we do other features. Certainly, teacher-fellows in all cohorts received extensive feedback from the instructional coach on their evolving ETPs. Their learning was supported in labs by PIs and research mentors. We have self-reports from teacher-fellows about the input and feedback they received throughout the program, but not documentation.
Teacher-fellows were asked to keep private Reflection Journals, in which they reflected on prompts such as: How is Design Thinking similar to and different from other teaching frameworks you have used? What are the strengths and challenges of implementing Design Thinking in your class? How might the strengths of distance learning be leveraged in curricula that involve Design Thinking? Journals were not collected or reviewed by research mentors, the instructional coach, or the Monday PD facilitators.
Teacher-fellows met each week with their instructional coach to develop and refine their ETPs. In the process of developing their ETPs, teacher-fellows were called upon to draw from a range of sources to produce a curricular module that other teachers would be able to implement. The ETP was required to identify the focal content and supporting pedagogical practices that are core to their ETPs, drawing from the Next Generation Science Standards (NGSS), California Content Standards, California Standards for the Teaching Profession, Math Common Core Standards, Career and Technical Education Standards, National Engineering and Technology Standards, National Board Standards from the National Board for Professional Teaching Standards, or the 21st Century Skills. Working with their instructional coach, teacher-fellows were required to identify up to three measurable learning objectives and the assessment tools teacher-fellows will use to assess the students' ability to apply or demonstrate understanding of the content, concepts, and practices. Teacher-fellows valued the feedback and opportunities they had to reflect as they developed their ETPs: The one-on-one meetings with our coach were the most effective, as they addressed specific requirements of my ETP. I received valuable feedback throughout the process, as well as at the end, which helped to elevate my ETP. I believe the suggestions during the process were the most essential, as they helped prevent unnecessary, In addition to the feedback and reflective time built into the development of the ETPs, teacher-fellows were prompted to reflect on their ETP exchange sessions, during which two teacher-fellows read and discuss one another's ETPs. The exchange and discussion is supported by the teacher-fellows' use of a feedback form with key prompts, to generate suggestions for improvements or enhancements. In the spring following their summer experience, after they have implemented their ETPs, teacher-fellows are required to complete Impact Reports, in which they reflect on the experience of delivering their ETP, what they learned, and the potential impacts on their students. Program leaders collect Impact Reports, but teacher-fellows do not receive feedback on them. Although teacher-fellows receive extensive input and feedback throughout the designing of their ETPs, there is no mechanism in the Stanford summer program by which teacher-fellows receive direct feedback on implementing their lessons. Therefore, we conclude that this feature of high-quality teacher professional learning was only partially evident in the Stanford RET program. The RET summer experience is eight weeks in duration, with teacher-fellows expected to participate for eight hours each day, for a total of 320 hours. Translated into continuing education credits (10 hrs = 1 credit), that is equivalent to 32 continuing education credits. In addition, teacher-fellows follow-up on the summer experience during the academic year. In the spring following their summer experience, after they have implemented their ETPs, teacher-fellows complete Impact Reports (mentioned above), which are designed to link teachers' summer experiences to their classroom practice.
C3 teacher-fellows reported spending on average between 15-100 hours on their ETPs, with a mean of 41. This compares to C2 teacher-fellows who reported 6-100 hours with a mean of 27. We believe that the difference is attributable to the fact that teacher-fellows experiencing virtual delivery had to expend more time and effort researching information as well as organizing their presentations for remote (vs. classroom) delivery. Teacher-fellows in both cohorts spent substantial amounts of time preparing their ETPs, which, as discussed above, involved extensive coaching, support, and reflection-far more than is typically included in most teacher professional learning experiences.

DISCUSSION
Our first research question is: To what extent did the RET program incorporate Darling-Hammond et al.'s (2017) seven research-based characteristics of effective teacher professional learning? Evidence suggests that the Stanford RET program included all seven characteristics of effective teacher professional learning. Six characteristics were fully integrated; reflection on instructional design was included, but reflection on implementation was not. Lab placements for 35 hours each week ensured a heavy focus on disciplinary content. Lab experiences involved active learning almost entirely. Teacher-fellows collaborated extensively with lab partners, PIs, the instructional coach and sometimes with one another to complete lab tasks and develop their ETPs. Monday PD sessions, both virtual and in-person, modeled effective teaching practice. The RET teacher-fellows received extensive support from lab PIs and research mentors, as well as the instructional coach in understanding and participating in lab projects and developing their ETPs. Finally, the 5-days/week, 8-week program (equivalent to 32 continuing education credits), was of substantial duration.
Our second research question asked: To what extent do immediate outcomes of 100% virtual program delivery compare to outcomes of in-person program delivery? Interviews and exit surveys show no significant or meaningful differences in program outcomes between in-person and virtual cohorts. Comments from C3 teachers were mostly positive, even among returning teacher-fellows. Inevitably, the suc-cesses were accompanied by at least some sense of loss: As it was virtual, I did not feel that I contributed as much as I could have, had we been in person, and that is just me personally. When I did get a chance to visit the lab I was so impressed and wished I had had the opportunity to see it in action! If there could be a camera or something set up to help those who are working virtually-if you are going to offer this to teachers again virtually-I think that would be a great help. (C3 Teacher-Fellow) Comments from returning teacher-fellows who had experienced in-person and remote delivery include: Overall, the data across three cohorts show that the move to virtual programming did not significantly influence or diminish the desired outcomes of the program.
Having said that, the nature of the experiences of in-person delivery compared to remote delivery differed considerably. Teacher-fellows in C1 and C2 interacted directly with laboratory equipment, and worked one-on-one next to graduate student mentors and PIs. C3 teacher-fellows, who could not work in labs, were more focused on data analysis, auxiliary research and converting technical content into classroom lessons, often for remote delivery. However, all cohorts, including C3, reported extensive hands-on learning. C3 teacher-fellows felt engaged with laboratory culture, even though returning teacher-fellows, who could compare remote and in-person delivery, acknowledged that virtual-delivery wasn't the same. Our data show that the nature of the experience varied based on the mode of program delivery, but the benefits and outcomes--including actively collaborating with research staff, engaging science and engineering content, and learning to communicate technical content--were achieved for all cohorts.
We hasten to point out that the summer 2020 program included only a limited sample of eight teacher-fellows, due to the number of faculty willing to support teacher-fellows in a remote lab environment on short notice. Given the selectivity of the PIs and prior RET experience of four of the eight teacher-fellows, readers may speculate that the success of remote delivery in 2020 was anomalous. We don't think so. Communication strategies rarely implemented prior to COVID-19 have become routine. And, in retrospect, COVID-19 mitigations have not been as dire as they seemed initially. One scientist mentor said of remote program delivery, "It's a working style we had before COVID-19. We have written papers with people we have never met in person, only through Zoom interactions, and for us the transition to current times has been easy." In Summer 2021 (operating as we write this), Stanford RET includes 16 teacher-fellows and 13 PIs. The Stanford RET Director reports that 13 projects are entirely remote and three are hybrid (in-person and remote access). Exit surveys are expected to confirm the efficacy of remote program delivery.
Limitations. As mentioned above, the 2020 cohort was limited to eight participants, five of whom were returning alumni who were already familiar with the program goals and activities, unlike most teacher-fellows from 2018 and 2019. Second, summer 2020 fellows were placed in labs in which the PIs were experienced hosts of teacher-fellows and who were committed to experimenting with remote delivery of RET. Third, and most notably, while we can demonstrate that the Stanford RET program met Darling-Hammond et al.'s criteria of effective teacher learning, we have no evidence that embedding those characteristics improved classroom teaching practices and/or student achievement.

Implications: Challenges and Opportunities Presented
by Virtual Programming. The most obvious challenge of Stanford's 2020 lab-based, summer RET program was that no one was able to get into the labs! The obvious drawbacks were counterbalanced by substantial opportunities. In interviews and on exit surveys, teacher-fellows describe experiencing hands-on learning, being engaged in lab culture, and learning science and engineering content that they can integrate into their classrooms. Notably, remote delivery broadened participation geographically, supporting teachers for whom distance and time away from home and children would have prevented participation. This study contributes to research findings that suggest the quality of teacher professional learning does not depend on the modality of its delivery. RET program solicitations embed several of the characteristics of effective teacher professional learning opportunities outlined by Darling-Hammond et al. (2017); yet, little research has been funded to explore whether RET summer experiences change teachers' instructional behaviors or result in improved student out-comes. Because the components of RET programs are, by design, similar across contexts, the RET program offers a rich opportunity to explore the effects of the summer programs in teachers' classrooms.
This historical moment provides an opportunity to compare classroom outcomes based on in-person vs. virtual program delivery. If engineering is going to be taught meaningfully in K-12 schools, substantial teacher professional learning will be critical for boosting science teachers' capacity to teach engineering content and methods. The National Academies of Science, Engineering, and Medicine (2020) found little research connecting teachers' professional learning to classroom teaching behavior or student outcomes. RET programs are a promising venue for exploring and patching that gap.

AUTHOR INFORMATION Corresponding Author
Gary Lichtenstein. gary@QualityEvaluationDesigns.com

Author Contributions
The manuscript was written through contributions of all authors. All authors have given approval to the final version of the manuscript.