Providing Transferable, Professional Skills for the Next Generation of Scientific Professionals through an Outreach Opportunity

Today, scientists increasingly bear the responsibility of sharing scientific results with the public. To help meet this challenge, there is a need to provide scientists with training in science outreach and education. This kind of professional development would benefit both scientists and the public if scientists could learn these skills while simultaneously creating resources that can be shared with the larger community. To this end, we have developed “Annotator Professional Development,” an interactive professional development opportunity designed to introduce interested graduate students, postdoctoral fellows, and STEM professionals to the basic principles of science education through participation in an educational outreach opportunity. Here we present details about this project including data suggesting that participants begin to comprehend how outreach skills can be valuable in a variety of STEM careers. We also discuss the feasibility and potential of expanding the program. INTRODUCTION In today’s world, there is an urgent need to provide professional development for scientists in science outreach and education. Despite this need, there are very few opportunities in educational outreach that provide such training for scientists while simultaneously benefiting the larger science, technology, engineering, and math (STEM) community. While professional development in the transferable skills of science outreach and education training will benefit all scientists, the graduate students in STEM fields represent an especially valuable target audience as they have their entire careers ahead of them. In addition, the career development opportunities offered during doctoral training programs in STEM have not kept pace with the dynamic shifts in the career paths of their graduates, raising questions as to whether the current system of graduate education is adequately preparing students to enter a 21st-century job market (National Academies of Sciences, Engineering, and Medicine [NAS], 2018; Denecke et al., 2017; Gibbs et al., 2015; Alberts et al., 2014; Fuhrmann et al., 2011). Why are opportunities for graduate students in career and professional development services not the norm at most institutions (Fuhrmann et al., 2011; Ashby and Maher, 2018; Ritchie et al., 2018)? One reason may be the perception of a negative trade-off between time spent on research and time spent on professional development. There is a misconception that providing graduate students with additional experiences will dilute the core elements of the Ph.D., add additional commitments to already stressed students, and increase the time to graduation (NAS, 2018). In fact, the opposite may be true, with several studies showing that career development activities do not negatively impact research training or productivity (Fuhrmann et al., 2011; Shortlidge and Eddy, 2018; Feldon et al., 2011). This suggests that the absence of career development opportunities as a standard feature in graduate training is a missed opportunity, as diversity in graduate training opportunities can result in graduate students becoming better scientists and more desired job applicants (Loshbaugh et al., 2011). With 82% of STEM graduates with Ph.D.s entering a non-academic career path, it becomes imperative to ensure that this population remains engaged with STEM education and outreach initiatives (Science and Engineering Indicators, 2018). In fact, we should be training all those holding STEM Ph.D.s to become lifelong advocates for STEM education and outreach. One way to develop such advocates is to provide graduate students with training at the intersection of education outreach and discipline-based education research (DBER) – the relatively new field of interdisciplinary research that combines the expertise of scientists and engineers with methods and theories that explain learning (National Research Council [NRC], 2012). DBER “investigates learning and teaching in a discipline from a perspective that reflects Transferable skills Ritchie Vol. 2, August 2019 Journal of STEM Outreach 2 the discipline’s priorities, worldview, knowledge, and practices” (NRC, 2012). Results from DBER studies have led to meaningful improvements in student learning and participation in STEM disciplines, although many challenges and unanswered questions still remain in this nascent field (Henderson et al., 2017). DBER refers to teaching practices that are informed by objective evidence, such as educational research or metrics of student performance, and DBER principles are derived from specific findings from this type of research (e.g. active learning, inquiry-based labs, and aligning course work with laboratory work). DBER training can be attractive to graduate students, as the skills are transferable and applicable to students interested in both traditional and non-traditional scientific careers. For example, a recent study found that graduate students participating in teaching activities reported improvements in their writing skills, communication skills, and time management and organizational skills (Jordan and Howe 2018). Another study showed that graduate students who engaged in teaching showed more improvement in their abilities to generate testable hypotheses and design valid experiments than did students who did not engage in teaching (Feldon et al., 2011). Additionally, undergraduate STEM students who explored teaching, but did not ultimately enter the profession, showed positive and lasting changes in interest in future STEM-related volunteering in public schools and in intentions to support STEM outreach efforts in the workplace (Whang-Sayson et al., 2017). These data may suggest that being introduced to DBER through an outreach opportunity, at both the undergraduate and graduate level, results in more well-rounded STEM professionals. It also suggests that investments made in training graduate students in DBER will not only benefit these students as they start their careers, but will also benefit the numerous individuals and institutions they will interact with throughout their careers as STEM ambassadors. Similar research relating to skills gained through outreach experiences is limited and through the program discussed in this article we are hoping to fill this gap (Clark et al., 2016). The scientific literature contains many examples of how students can learn science through participation in an outreach event. Building off of this knowledge, we hypothesized that graduate students would also be able to learn science, in this case DBER principles, as they participated in an outreach event (related to Science in the Classroom, described further below). This report describes an exploratory pilot program of “Annotator Professional Development (APD),” an interactive professional development opportunity designed to introduce interested graduate students, postdoctoral fellows, and STEM professionals to the basic principles of DBER through an activity that creates an educational resource, as the participants develop annotated research papers that are made freely available (Figure 1). Because the participants are introduced to DBER principles before they develop this educational resource, they also acquire skills that will be important for their future employment. While this Figure 1. An overview of the structure of APD training. Here, we define acronyms and show the relationships between SitC, APD, and DBER. Transferable skills Ritchie Vol. 2, August 2019 Journal of STEM Outreach 3 program was framed within the context of graduate school, we acknowledge that professional development is continuous throughout one’s scientific career and anticipate interest from a range of participants.


INTRODUCTION
In today's world, there is an urgent need to provide professional development for scientists in science outreach and education. Despite this need, there are very few opportunities in educational outreach that provide such training for scientists while simultaneously benefiting the larger science, technology, engineering, and math (STEM) community. While professional development in the transferable skills of science outreach and education training will benefit all scientists, the graduate students in STEM fields represent an especially valuable target audience as they have their entire careers ahead of them.
In addition, the career development opportunities offered during doctoral training programs in STEM have not kept pace with the dynamic shifts in the career paths of their graduates, raising questions as to whether the current system of graduate education is adequately preparing students to enter a 21st-century job market (National Academies of Sciences, Engineering, and Medicine [NAS], 2018;Denecke et al., 2017;Gibbs et al., 2015;Alberts et al., 2014;Fuhrmann et al., 2011). Why are opportunities for graduate students in career and professional development services not the norm at most institutions (Fuhrmann et al., 2011;Ashby and Maher, 2018;Ritchie et al., 2018)? One reason may be the perception of a negative trade-off between time spent on research and time spent on professional development. There is a misconception that providing graduate students with additional experiences will dilute the core elements of the Ph.D., add additional commitments to already stressed students, and increase the time to graduation (NAS, 2018). In fact, the opposite may be true, with several studies showing that career development activities do not negatively impact research training or productivity (Fuhrmann et al., 2011;Shortlidge and Eddy, 2018;Feldon et al., 2011). This suggests that the absence of career development opportunities as a standard feature in graduate training is a missed opportunity, as diversity in graduate training opportunities can result in graduate students becoming better scientists and more desired job applicants (Loshbaugh et al., 2011). With 82% of STEM graduates with Ph.D.s entering a non-academic career path, it becomes imperative to ensure that this population remains engaged with STEM education and outreach initiatives (Science and Engineering Indicators, 2018). In fact, we should be training all those holding STEM Ph.D.s to become lifelong advocates for STEM education and outreach.
One way to develop such advocates is to provide graduate students with training at the intersection of education outreach and discipline-based education research (DBER) -the relatively new field of interdisciplinary research that combines the expertise of scientists and engineers with methods and theories that explain learning (National Research Council [NRC], 2012). DBER "investigates learning and teaching in a discipline from a perspective that reflects the discipline's priorities, worldview, knowledge, and practices" (NRC, 2012). Results from DBER studies have led to meaningful improvements in student learning and participation in STEM disciplines, although many challenges and unanswered questions still remain in this nascent field (Henderson et al., 2017).
DBER refers to teaching practices that are informed by objective evidence, such as educational research or metrics of student performance, and DBER principles are derived from specific findings from this type of research (e.g. active learning, inquiry-based labs, and aligning course work with laboratory work). DBER training can be attractive to graduate students, as the skills are transferable and applicable to students interested in both traditional and non-traditional scientific careers. For example, a recent study found that graduate students participating in teaching activities reported improvements in their writing skills, communication skills, and time management and organizational skills (Jordan and Howe 2018). Another study showed that graduate students who engaged in teaching showed more improvement in their abilities to generate testable hypotheses and design valid experiments than did students who did not engage in teaching (Feldon et al., 2011). Additionally, undergraduate STEM students who explored teaching, but did not ultimately enter the profession, showed positive and lasting changes in interest in future STEM-related volunteering in public schools and in intentions to support STEM outreach efforts in the workplace (Whang-Sayson et al., 2017). These data may suggest that being introduced to DBER through an outreach opportunity, at both the undergraduate and graduate level, results in more well-rounded STEM professionals. It also suggests that investments made in training graduate students in DBER will not only benefit these students as they start their careers, but will also benefit the numerous individuals and institutions they will interact with throughout their careers as STEM ambassadors. Similar research relating to skills gained through outreach experiences is limited and through the program discussed in this article we are hoping to fill this gap (Clark et al., 2016).
The scientific literature contains many examples of how students can learn science through participation in an outreach event. Building off of this knowledge, we hypothesized that graduate students would also be able to learn science, in this case DBER principles, as they participated in an outreach event (related to Science in the Classroom, described further below). This report describes an exploratory pilot program of "Annotator Professional Development (APD)," an interactive professional development opportunity designed to introduce interested graduate students, postdoctoral fellows, and STEM professionals to the basic principles of DBER through an activity that creates an educational resource, as the participants develop annotated research papers that are made freely available (Figure 1). Because the participants are introduced to DBER principles before they develop this educational resource, they also acquire skills that will be important for their future employment. While this program was framed within the context of graduate school, we acknowledge that professional development is continuous throughout one's scientific career and anticipate interest from a range of participants.

BACKGROUND
Annotated Primary Literature and Science in the Classroom. Annotated primary scientific literature is a pedagogical tool used to introduce undergraduate students to primary scientific literature (Kararo and McCartney, 2019) that builds off of the early work on Adapted Primary Literature by Yarden, Norris, and Phillips (Yarden et al., 2015). Science in the Classroom (SitC) (www.scienceintheclassroom.org/) is an example of annotated primary scientific literature, that adds layers of additional information onto the original text of primary scientific literature, ultimately making primary scientific literature more accessible to a wider audience ( Figure  2). An online tool, called the Learning Lens, allows users to select highlighted parts of the accompanying piece of primary scientific literature that fall into different, color-coded categories (e.g., glossary, previous work, author's experiments, results and conclusions, news and policy links). For example, a user clicking on a yellow text selection, corresponding with the glossary category, will find the definition and more information relating to that particular section of text. With this level of extra context and background, students, specifically at the undergraduate level, can see how the authors identified a question, collected and analyzed data, and pro-posed the next question(s).
Since launching in 2012, SitC has had over half a million visitors, trained over 150 annotators, and has published over 100 annotated articles. Through a series of outreach presentations and intensive workshops, SitC has created a community of practice of educators and volunteers who are invested in improving resources for bringing primary scientific literature into the classroom. SitC has worked with the Howard Hughes Medical Institute (HHMI) BioInteractive, iBiology, The Library of Congress, The Smithsonian Institution, and other institutions to create interdisciplinary and multimedia content that is relevant to significant modern scientific issues. In addition, introductory research on the use of annotated primary scientific literature in undergraduate classrooms has demonstrated that students respond positively to annotated primary scientific literature (Kararo and Mc-Cartney, 2019). Taken together, these data suggest SitC is a successful STEM outreach project.
Participants contribute to the SitC outreach project through writing and designing the annotations. In previous iterations of this training (known as SitC training), 100% of participating graduate students have stated that they listed participating with SitC on their curricula vitae (CVs) and 82% agreed or strongly agreed that SitC training provided additional credentials for their CVs (McCartney et al., 2018). With this level of interest and satisfaction in SitC training, and the clear indication that participants are treating the experience as a career-enhancing opportunity, it was hypothesized that Annotator Professional Development could be amplified to also serve as an introduction to DBER.
Annotator Professional Development. The SitC training was modified and a pilot version of the current training, which will be referred to as Annotator Professional Development (APD), was offered between April 2017 and July 2018. This program evaluation investigated how participants responded to the training itself and the DBER content and outreach knowledge they gained, related to the three goals of APD: Training: Introduce participants to the general principles of DBER.
Outreach: Enable participants to develop resources to be used by the STEM community.
Impact: Facilitate adoption of DBER principles in a variety of future STEM career paths.
In the work presented here, we seek to begin to assess how well APD accomplished these goals through a formative program evaluation guided by the following questions:

1)
What do participants already know about DBER?
2) How do participants engage in/envision DBER to be implemented at their current institution?
3) What DBER topics are of interest to participants?

4)
Can participants understand how to apply DBER principles under certain learning situations?
APD was an iterative and reflective training. Over a series of six videos, on a time scale convenient to the participants' schedules, they were introduced to various aspects of DBER and asked to reflect on if and how these principles were applied to their own schooling. Participants then considered how to apply these same DBER principles moving forward in their careers. Upon successful completion of the training videos, each participant annotated a scientific article that will become a part of the freely-available body of SitC resources. Participants will also have a tangible product they can list on their CVs or discuss during an interview.

MATERIALS AND METHODS
APD is freely-available and offered via the SitC website (http://www.scienceintheclassroom.org/volunteer). For this program, all interested volunteers, at the graduate level and above in their STEM careers, were invited to participate. The American Association for the Advancement of Science (AAAS), which hosts and supports SitC, advertised this opportunity using relevant email listserves and social media accounts. This version of APD was modeled on the best practices established during previous iterations of the annotator training (McCartney et al., 2018). Training consists of six modules covering the following topics titled: (1) What is SitC and what to expect from this training, (2) An Overview of DBER, (3) Academic Language and Science Communication, (4) Science Education Frameworks and Standards, (5) Primary Literature as an Educational Tool, and (6) Writing Annotations. As participants progress through the training, they are required to complete an accompanying worksheet that includes several reflective questions for each video. Collectively, each training video and the accompanying worksheet questions hereinafter will be referred to as a module. An overview of the videos, worksheets, and readings is provided in detail below and summarized in Table S1.
Module Design. Because the target participants of APD are graduate students, the aim was to make APD similar to a course they would be taking within their individual disciplines. To achieve this, APD modules were designed using an adapted version of Bloom's Taxonomy of learning domains: remember, understand, apply, analyze, evaluate, and create (Anderson et al., 2001;Krathwohl, 2002;Crowe et al., 2008). In agreement with Bloom's, APD modules do not emphasize memorization and recall, or lower-order cognitive skills, but instead focus on applications of knowledge and critical thinking, or higher-order cognitive skills that require deep conceptual understanding (Zoller, 1993).
Connecting to Bloom's taxonomy allowed the training to follow the same natural progression that most graduate programs use to train their students to become experts. Most students learn the fundamental theories of their discipline in their coursework along with how to use resources to advance their understanding of related fields (e.g. SciFinder, Web of Science, etc.) (SciFinder, 2019, Web of Science, 2019). Students also typically learn how to analyze data through reflective processes (e.g. group meeting, journal club) and work on original research projects. This general approach provided a familiar framework for training participants.
Extensive research has shown that active-learning methods improve student achievement in college STEM courses (Freeman et al., 2014). Therefore, aspects of active learning were also incorporated into APD modules, although this was challenging in an online environment. Drawing on evidence from the Physics community, "guided worksheets" (described further in "worksheet design") were developed to accompany each module as a way to help participants keep pace with and focus on the most relevant content presented (Sujarittham et al., 2016). In addition, engaging students' metacognitive skills has been shown to increase the effectiveness of online courses (Cacciamani et al., 2012). In line with this, the worksheets contained within each module had students contemplate their ideas and take time to reflect on the DBER strategies presented. rizing a DBER paper and a funded DBER project. Session 3: Several reflective questions on the participants' experiences with academic language and reading primary scientific literature. Session 4: Reflective questions on how the participants were taught science in their own education, how each set of frameworks and standards relate/connect to each other, and how the frameworks and standards relate to DBER principles. Session 5: Reflective questions on how the three models for teaching with primary literature compare and contrast to each other and on how all of these models relate to DBER principles. Session 6: In lieu of reflective questions, participants are asked to practice writing annotations.
Reading selections. Suggested readings are provided to accompany each video module as a way to provide additional background information to participants and to increase involvement in the training. Selected readings are described in Table 1.
Data collection. In order to target higher-order cognitive skills, a qualitative approach to data collection was followed, where the trainees' previous scientific experiences in their institutions, classrooms, and their own educations were used to understand a typical graduate student's familiarity with DBER. This allowed for the collection of a rich data set while engaging trainees in reflective and evaluative processes based on the information they were learning through the training (Lemons and Lemons, 2013). As a result, most of the collected responses were free-response text.
Data-collection was a two-step process. Participants completed a pre-training questionnaire where they were instructed to create a unique ID as a way to de-identify data.
In an effort to separate the research team from the training process, and to ensure privacy, participants were asked to enter their unique ID when completing the worksheets in order for the research team to connect their worksheets to their questionnaire. The implementation of using a unique ID was not entirely successful. Nine of the twenty-four participants did not remember their ID between completing the questionnaire and beginning the training and, as a result, the research team was unable to collect demographics on all twenty-four participants. Therefore, only fourteen demographic sets were officially recorded. However, AAAS staff work directly with all annotators and only allowed qualified participants (graduate-level and above) to continue with the training. Of the fourteen participants with an official demographic record, four were graduate students, three were postdoctoral fellows, five identified as being a teacher or instructor, one identified as a faculty member, and one identified as a non-profit professional. While training is done individually and at the participant's own pace, feedback suggests it takes approximately 6 hours to complete the videos and accompanying worksheets. While this program is aimed specifical-Video content. Videos were recorded at AAAS in the Fall of 2016. Videos are hosted on YouTube and are freely available. Video scripts and slides were developed with input from the SitC Advisory Board and colleagues who are experts in teaching and learning. Feedback and advice from previous participants who completed earlier versions of annotator training was also considered (McCartney et al., 2018). Video content is as follows: (Session 1, 15 minutes in length) What is SitC and what to expect from this training. This session includes an introduction to SitC staff, a general overview of SitC and how SitC connects to STEM education, how an annotated research paper is developed, and what to expect during this training. (Session 2, 11 minutes in length) An Overview of Discipline-based Education Research. This session includes an overview of DBER, how DBER principles connect to annotations, where to find research papers centered on DBER, and how to fund DBER projects (National Science Foundation [NSF], National Institutes of Health [NIH], private foundations). (Session 3, 9 minutes in length) Academic Language and Science Communication. This session includes a discussion of academic language, how academic language relates to annotations, reflection on personal experiences with primary literature as an undergraduate student, and the difference between a novice and an expert reader and how this relates to the structure of a research paper. (Session 4, 6 minutes in length) Science Education Frameworks and Standards. This session includes a discussion on why we have learning frameworks and standards, a comparison of several different learning frameworks and standards (Advanced Placement, A Framework for K-12 Science Education, and Vision and Change in Undergraduate Biology Education), and how these frameworks and standards relate to writing annotations. (Session 5, 9 minutes in length) Primary Literature as an Educational Tool. This session includes an introduction to other examples of how to use primary literature as a tool for promoting DBER principles and as a way to connect to learning frameworks and standards, and how each of these methods connects to annotations. (Session 6, 20 minutes in length) Writing Annotations using DBER Principles. This session includes frequently asked questions related to the annotation process, a demonstration of each annotation category, and examples of what a good annotation looks like.
Worksheet design. Worksheet questions are summarized as follows. Session 1: Reflective thoughts on how annotations are similar/different to how participants themselves markup scientific literature. Session 2: Reflective thoughts on how participants themselves learned science during their own education and how this relates to DBER principles, how DBER principles are (or are not) practiced in the participant's institution, and participant experience with reading DBER literature. Participants are also tasked with finding and summa-ly at graduate students, its relevance and usefulness to other members of the STEM community is acknowledged.
Because this was a pilot program, and the research team had no knowledge of the previous experiences of the participants, a qualitative research method was chosen as a way to further understand how participants engaged with the training modules. Data were collected from annotators participating in training between April 2017 and July 2018 (Florida International University [FIU] IRB approval # IRB-17-0109). Data were collected through pre-and post-training questionnaires as well as the worksheets completed during the training. Participant survey responses used a unique ID code so that data could not be traced back to any individual person. AAAS staff collected all data and any identifying information was removed before they were given to the research team. The research team was completely separate from AAAS during this formative program evaluation.
Data Analysis. The researchers read all of the short answer question responses and independently created lists of the different perceptions, attitudes, and opinions contained within these responses using thematic analysis (Braun and Clark, 2006). The researchers compared the initial findings from the short answer questions and developed a comprehensive code book. The researchers then used the codebook to independently re-code the short answer question responses, during which the researchers reconvened as needed to discuss and further define responses that were unclear. All coding was conducted using NVivo software (version 11.4, QSR International). Adobe Analytics was used to determine user interactions with annotated papers. The analysis window was set for August 2017-August 2018. Total clicks, page views, and unique visitors values were recorded.

RESULTS
APD began with module 2.1: A presentation of DBER background and principles. Participants were asked to reflect on whether or not DBER principles had been used during their own science education, with nearly half of the participants claiming no prior exposure to DBER techniques (n=11 out of 24). However, short answer responses revealed that many participants would have liked to have had knowledge of DBER available to them and they often provided examples of how DBER would have enhanced their own experiences (Figure 3) This was provided as a brief overview of DBER. Participants were encouraged to read the entire report, however only the executive summary was required. These were provided as an introduction to different frameworks and standards in STEM education. Participants were encouraged to read the entire reports, however only chapters 4 and 5 were required for Framework and only pages xiv-xv and chapters 1-3 were required for Vision and Change.
https://apcentral.collegeboard.org/courses/resources/science-practices  the several answers given, the most popular approach was through the creation of a new journal club (42%). A representative response is shown (additional responses can be found in Section S1): "I think it might be fun and instructive to have a voluntary group within my department read DBER research from different disciplines. Maybe one person reads about DBER research in geology, another in physics, and so on. Then, the individuals in the groups could share approaches of these disparate disciplines, and they could consider together how these approaches are transferable to their own discipline." One participant touched on the objective of this program and indicated the potential benefits of incorporating DBER into the graduate curriculum: "...I do believe using DBER can be useful not only when teaching graduate students but also in training them as teaching assistants. Encouraging the use of DBER can be done through first running surveys on how graduate students are performing as TAs and if they are reaching their audiences, implementing training for TAs which includes reading DBER publications and requiring graduate students that work as TAs [...] to present their research to teachers and high school students, and to implement at least one of the evidence-based practices that have been supported by DBER." A crucial aspect to understanding DBER requires participants to learn where they can find resources on theoretical frameworks, previous and current research, and commentary on the field. Therefore, the next part of the training (modules 2.5 and 2.6) focused on introducing annotators to the DBER literature. Participants were introduced to the Education Resources Information Center (ERIC) database, with 83% responding that they had not used ERIC prior to this training ( Figure S1A). Additionally, only 25% of participants acknowledged that they currently read and/or monitor at least one DBER journal ( Figure S1B). Several participants included the titles of DBER journals they followed, indicating that those participants who were familiar with DBER journals seemed interested in staying current with the DBER community ( Figure S1C).
The training continued to advance participants' critical thinking and engagement with DBER principles through two additional assignments. At the core of each assignment was introducing participants to the diversity of DBER, including various scientific disciplines, student populations, and learning objectives being considered. First, participants were tasked with using the ERIC database to find and summarize a DBER article. This assignment served to have participants explore a DBER article as an introduction to how DBER studies are designed and implemented. Using thematic analysis, articles selected by participants were characterized into one of the categories listed in Table 2, as an exploratory analysis of understanding what kinds of DBER studies participants were interested in.
Similarly, participants were tasked to review recently funded projects listed on the NSF Improving Undergraduate STEM Education (IUSE) website. This assignment served as an introduction to participants on how DBER research studies are funded and how DBER ideas are presented in a research framework. Participants were then asked to go a step further and describe a DBER project they themselves would be interested in carrying out. Thematic analysis of proposed project categories are summarized in Table 2, again as an exploratory analysis of understanding what kinds of DBER studies participants were interested in. The following is a representative participant reflection highlighting the higher order cognitive skills taking place. Additional participant responses can be found in Section S1.
"Just looking at the California abstracts, there were only four out of 106 proposals that had anything to do with ocean research and none are available for Northern California. Considering the Bay Area's proximity to the ocean and the San Francisco Bay in particular, it would be worth it to create an oceanography and/or coastal science program to encourage high schoolers and undergraduates to experience ocean research. Perhaps in a collaboration with some of the institutes in the area, students could have the experience going out on a research vessel and collecting water samples or sediment samples. It might even be possible to create a program where students monitor the same areas each year in order to create a timeseries record of variability in coastal ocean nutrients or phytoplankton, thus hitting both outreach with diverse students and also science monitoring of environmental variables which could be useful in producing peer-reviewed science, possibly with the help of some of the students involved in the program. Such a program would be training the next generation of geoscientists and oceanographers." Module 4.2 focused on integrating DBER principles into the annotation process. Specifically, the training was designed to demonstrate how current science principles focus more on teaching scientific process and analytical skills compared to previous generations where there was a strong focus on memorization. In addition, the trainees were shown how the formation and evolution of DBER created a paradigm shift in how educators are now being trained. When asked "How do you remember learning science in K-12?" several participants indicated that their strongest recall were for courses that incorporated some form of an active learning/DBER approach: Other participants indicated that their teacher's creative pedagogical techniques shaped their student experience: "I actually remember learning science in K-12 as a way to study the world, as a way to ask questions and try to find answers. I fell in love with science when we were learning about states of matter and my teacher put a petri dish of colored water on the overhead projector and asked us to try to figure out why the water "disappeared". Making that connection for myself, that the water molecules got warm and therefore wanted to be further apart and became a gas was a big influence in my life." Participants were then asked to evaluate three different learning frameworks (Advanced Placement, A Framework for and Vision & Change in Undergraduate Biology). Specifically, participants were asked to evaluate the similarities between the learning frameworks and reflect on their application at all learning levels. Most participants found the learning frameworks to be fairly similar and further elaborated on the following major themes (Table 3): All learning frameworks were found to: • have the same goals, just focused on different population • place an emphasis on skills compared to content • emphasize understanding the scientific process Table 3. Thematic analysis of participant responses to comparing and contrasting different learning standards and frameworks.
The following participant quote summarized the three major themes in one response: "The frameworks and strategies complement each other, being intended for different populations of students while having similar goals. The three sets of practices/competencies are almost the same. They emphasize learning scientific skills over facts and are intended to lead students towards an understanding of the process of science and how science relates to the world in which they live. They are inclusive, meant to educate the general population rather than just college-bound students (Framework)

or only science majors (Advanced Placement [AP] and Vision and Change)."
Ultimately, this module was designed to engage participants in understanding the connection between different learning frameworks and how those learning frameworks relate to the types of annotations they were preparing to develop. The following is an example of how participants interpreted this connection: "I will refer to these learning standards when annotating texts --I really liked how the Vision and Change table articulated and organized them. For example, I will explicitly highlight connections between concepts when writing Learning Lens excerpts. In designing educator guides, I will avoid "regurgitation"-type questions and ask more open-ended questions that invite student debate and encourage them [to] search for missing evidence or faulty arguments. The point is not to be right or wrong, but instead to practice reading, interpreting, and evaluating scientific studies. (E.g., what are the criteria required to adequately demonstrate a concept? Did the authors show sufficient evidence to meet those criteria? How would you design experiments for the future tests they recommend? Are their conclusions valid outside of their study? How might their choice of methods have biased their results?)" Module 5 had participants describe how learning frameworks connect to primary scientific literature, and, specifically, how annotated primary scientific literature is a way to present students with the practices of science. Additionally, two other evidence-based methods for using primary scientific literature as a teaching and learning tool (C.R.E.A.T.E and Figure Facts) were presented as research papers, effectively simulating a type of DBER journal club. Participants were asked to evaluate these three different approaches to teaching with primary scientific literature and consider a teaching situation (specific classes and/or specific learning goals) where they, as the teacher, might choose one approach over another (for example, teaching a seminar to 20 junior/ senior biology majors versus teaching high school sophomores). Here, a participant was able to integrate the three approaches: "I would use SitC, CREATE (sic), Figure Facts in ascending order of education level. SitC would get students familiar with research and build their confidence to approach papers. For older students, CRE-ATE (sic) would then further develop critical thinking, in-depth analysis and personal ideas on what would be good follow-up research. Figure Facts would also be good for students at this stage or more advanced to train "prediction skills" and be able to deduce how to extract data from complex figures and what conclusions might be inferred from this data." Participants were then given the chance to use their understanding of DBER and primary scientific literature to annotate a scientific paper. Participants went through an iterative editing process with SitC staff until the level of detail/ explanation met SitC standards with the final resource posted on the SitC website.
Having their annotated papers published on the SitC website provides participants with a tangible product that can be added to a resume or CV, presented in a job talk, or discussed in an interview. Adobe Analytics was used to ensure that the annotated papers were being accessed by the community (Fig. 4). User data shows that these papers are in fact being utilized by SitC users, demonstrating that the training has benefits for both the participants and the larger scientific community.

DISCUSSION
APD is a freely available, interactive professional development opportunity, designed to introduce graduate students, in any scientific discipline, to the basic principles of DBER while participating in an outreach opportunity. Providing graduate students with opportunities like this will help prepare them for a lifetime of being a STEM ambassador, across varied career paths. The training included having participants apply, analyze, and evaluate DBER principles within their own scientific experiences as a way to show them how valuable these principles are, regardless of discipline or career path.
Previous work has shown that short training experiences such as this have had limited success in producing longterm changes (Schussler, 2015;Feldon, 2017). This training can be seen as an "on-ramp" for students to explore DBER and is not meant to be a comprehensive course. And yet, our data show promising results for APD serving as a general introduction to DBER that provides successful participants with new knowledge of DBER principles that can be applied across multiple career paths.
Results from the pilot round of APD are encouraging. Nearly half (n=11 out of 24) of the participants had claimed no prior experience with DBER as a scientific discipline, suggesting that this training was in fact reaching the target audience: STEM graduate students who would now be able to enhance their graduate training with a further understanding of DBER. Qualitative data collected during this module indicate that participants were able to define DBER as a scientific discipline and reflect on how implementing DBER principles can enhance science education. This has value outside of academic or education-focused careers; for example, most careers in science policy will need a basic level of understanding of advances in science education, as will most roles in scientific administration.
Most scientific fields have specialized databases where users can find resources on theoretical frameworks, previous and current research, and commentary on the field. Participants were shown how to use the ERIC database, but, more importantly, were taught how to choose keywords or search terms and how to read and interpret the returned list of research papers. Knowing how to access this specialized database, and knowing how to identify relevant DBER content within it, is a transferable skill that will serve many different kinds of STEM professionals.
A second benefit of active ERIC searches was to show participants the extensive nature of the DBER literature. Although the data indicated that some participants were already familiar with DBER literature, their list of journal titles was narrow ( Figure S1C). Being able to see returned results from a variety of journals across numerous disciplines allowed participants to begin to comprehend how diverse DBER can be. Having this kind of perspective on the vast research literature underlying science education will be valuable in a variety of STEM careers, including grant managers evaluating education/outreach projects, museum, zoo, or park staff, or anyone involved in science education reform efforts.
Participants were directed to several different funding agencies and private foundations that invest in DBER. The purpose of this exercise was two-fold. First, it is highly likely that a search in the NSF IUSE database is new to graduate students, in terms of both search terms and results returned, because although DBER is growing as a field, it is still a fairly new scientific discipline for grad uate students to study compared to other subdisciplines. Second, by emulating what most graduate students learn in their individual research labs, i.e. if one has a scientific idea that he or she wants to pursue then it is important to search for a way to fund it, the training reinforced the idea that participants should approach DBER ideas in the same way they would for any other scientific discipline. Participants were then asked to look forward and describe a DBER project that they would be interested in carrying out. Taken together, this module provided participants with much more than a funding search. Participants were shown where (and how) to search for current DBER projects that could result in either inspiration for their own research or joining an existing project as a collaborator. This kind of reasoning can of course be transferable to any future STEM career, with participants now exploring funding opportunities outside of traditional basic research disciplines.
APD considers science education at a much higher level than the classroom -likely a higher level than most graduate students have ever encountered. Participants were given three different learning frameworks and were asked to compare/contrast them to each other, to consider how different organizations within the larger umbrella of science education complement each other, and to begin to comprehend the larger process of translating best-practices discovered through DBER to a classroom setting. Qualitative responses confirmed that this module not only raised awareness about the different learning frameworks, but also resulted in participants being able to relate these learning frameworks to their future scientific careers. Involving three different learning frameworks is unusual, as most current professional development opportunities focus only on a single framework, thereby setting APD apart.
Participants' new knowledge of DBER principles was integrated with their new understanding of learning frameworks as they were asked to evaluate how primary scientific literature can be used as a tool for teaching and learning. Essentially, participants read authentic DBER literature similar to what would happen in a journal club. Results showed par- . 'Total clicks' is the total number of times a SitC user clicked on a link that brought them to that particular annotated paper. It is important to note that this number does not capture the number of times an annotation connected to the paper has been clicked, only the number of times visitors came to this particular annotated paper. 'Unique visitors' are the total number of new SitC users viewing the annotated paper for the first time (this number does not account for repeat visitors with a few exceptions such as different devices, cleared cookies, etc.) 'Total page views' is the total number of views the annotated paper has received from visitors (this number will factor in repeat views). Further information on user interactions can be found in the supporting materials.
ticipants considering factors such as class size, class level, class structure, and specific learning goals when developing their own methods for teaching with primary scientific literature. Having this kind of understanding of DBER principles will be applicable to any number of STEM careers. Data analysis between annotations written by successful graduates of APD, compared to previous iterations of annotator training, is currently underway.
APD is unique for three reasons. First, prior growth in DBER has mainly occurred within individual disciplines and not as a collaborative effort across disciplines (Henderson et al., 2017). In contrast, APD was designed to be cross-disciplinary in nature and available to any member of the STEM community. The data presented in this report comes primarily from participants with a biology background; however self-identified chemists and physicists are currently enrolled in the training. This biologist-heavy participation has been the norm since SitC originally launched annotator training in 2009 and it remains unclear why there is less interest from chemists and physicists.
The second unique characteristic of APD is that, to the authors' knowledge, very few training opportunities at the graduate level focus on building skills in the context of creating shared resources. In contrast to existing professional development programs centered on training scientists in DBER, each of whom may develop a curriculum or teaching resource specific for their own courses and/or student population, the annotated papers developed through this training are made freely available to the entire STEM community. These annotated papers can be used in a variety of teaching and learning opportunities across all STEM disciplines and student populations.
Finally, the participants engage with and reflect upon DBER principles that will also be useful for careers in science policy and administration and science communication and outreach. Previous research from the CalTeach undergraduate teaching training program supports this claim, as undergraduate STEM students who participated in CalTeach, but did not ultimately become teachers, retained positive and lasting changes in interest in future STEM-related volunteering and outreach efforts (Whang-Sayson et al., 2017). Future Work. Thematic analysis of qualitative data has helped to generate knowledge of a novice's baseline DBER understanding. Future work includes transforming these qualitative themes into a quantitative questionnaire that can be used to measure the level of DBER understanding that participants have before and after this training, in essence a way of measuring DBER literacy. Such a questionnaire will also be useful for many institutions and departments that are implementing similar trainings in DBER principles.
Other plans include interviewing successful participants of APD as a way to better understand what modules were the most helpful and/or which modules should be updated. Included would be a longitudinal follow-up with participants to determine the lasting benefits of APD and how they were able to take advantage of DBER principles in their different types of career paths. Specifically, we would like to know what professions our APD alumni ended up pursuing and how many ended up as teachers.
We are currently increasing dissemination of APD through presentations at large national conferences, such as the AAAS annual meeting and the American Chemical Society national meetings. Additionally, we are expanding other aspects of APD including a new training for developing educator guides to accompany annotated primary scientific literature.
Limitations. SitC Annotator training is not meant to be a comprehensive course covering all aspects of DBER research. Previous work has shown that short trainings, such as this, have limited success in producing long-term changes (Schussler 2015, Feldon 2017. However, there is also evidence showing that undergraduate students who explored teaching, but did not ultimately enter the profession, retain positive and lasting changes in interest in future volunteering in public schools and in intentions to support outreach efforts in the workplace (Whang-Sayson et al., 2017). We consider SitC Annotator Training to be more in the former category, serving as a means to introduce participants to DBER principles and have them actively reflect on how these principles can be integrated into their current work and future careers as scientists.

ASSOCIATED CONTENT
Supplemental material mentioned in this manuscript can be found uploaded to the same webpage as this the manuscript.

Author Contributions
The manuscript was written through contributions of all authors. All authors have given approval to the final version of the manuscript.