ABSTRACT

A. Apprenticeship UREs

The URE programs traditionally follow an apprenticeship model (7). Students typically meet and are mentored directly by the lab’s principal investigator (PI) or a graduate student. The typical number of undergraduate students in these programs is limited due to the high resource cost of individualized mentoring and teaching (12, 13). A survey of biomedical research labs indicates that only 40% of research faculty surveyed had had more than 10 undergraduate students over a 5-year period (14). In an apprenticeship-style program, students are closely mentored on a highly individual basis, allowing them to quickly learn both research methodology and durable skills, such as time management, communication, and scientific intuition skills (15). These programs excel in creating strong mentor–mentee relationships and providing hands-on learning opportunities for the students (15).

At research-focused institutions (R1s), graduate students play a significant role in enhancing the URE, while also developing a mentor–mentee skill set (11, 16, 17). Graduate students also benefit directly from mentoring undergraduate research students, because undergraduate students can assist in generating and preparing data for the graduate student’s dissertation (8). Through mentoring undergraduates, graduate students can become more adept in communicating science, project management, and other durable skills that will help them in future careers in any career sector. An important skill gained by graduate students in mentoring undergraduates is managing projects with multiple students working together who often have differing skill levels and interests (18). Allowing graduate students to mentor early in scientific careers also builds confidence in continued leadership roles (19).

Apprenticeship URE models in biochemical and biophysical research often face challenges due to limited resources, resulting in smaller-sized student cohorts (12). There are also increased barriers for students to enter a research environment, including a lack of defined short-term outputs, little to no peer mentorship, and difficulties transferring skills from a laboratory course to a research environment (7, 20, 21). Finances also present a barrier to undergraduate research; without funding opportunities, many students cannot afford to participate in summer research due to a lack of on-campus housing or semester research due to multiple obligations (22). In addition, mentoring is time intensive for graduate students. There is often limited communication regarding outside resources for improving mentoring skills (23), and rarely is time set aside to use those resources. The lack of a defined structure and trained mentors can lead to frequent changes in expectations, which may cause undergraduate researchers to feel frustrated and burned out (8).

B. CUREs

In contrast to the variable nature of traditional UREs, CUREs provide a highly structured path toward learning and applying research skills (24, 25). These courses integrate research into a set curriculum, often teaching broader research skills, in addition to laboratory techniques (25, 26). In CUREs, students ideate and perform novel research on a smaller scale, with several CUREs resulting in publications from the research (26, 27). CUREs are effective in retaining students in STEM and increasing student confidence in research techniques (26, 28–30), and CUREs also benefit by allowing a far greater number of students to participate in research, often having 15 to >100 students per course. CUREs promote greater interdisciplinary work, due to more systematic training and learning content (31, 32). The structured nature of CUREs allows instructors at other institutions or domains to easily adapt the curriculum and research experience to fit the learning objectives (26, 33).

Similar to traditional UREs, graduate students can also play a significant role in CUREs. Recent studies have shown that CURE graduate teaching assistants (GTAs) have a high impact on both the research outcomes and student outlook on research (34, 35). Many GTAs focus on durable skills, while others focus much more on practical technical methodologies, affecting students’ perceived confidence in the ability to perform scientific research (35). GTAs have also been shown to gain mentorship skills while assisting in CUREs, although there is a greater emphasis on technical problem-solving than mentoring durable skills (1). CUREs can often serve as recruitment for the faculty instructor’s lab, allowing graduate students to find which researchers they can form the strongest mentor–mentee relationships with to join more traditional URE (36).

Many CUREs face challenges in balancing scales to promote long-term student research outcomes (37). Recent surveys have indicated that many of the skills learned in early CURE courses may not be sustainable for the entirety of a student’s degree program (38). There are also challenges in maintaining adequate background and theoretic knowledge, while keeping the course accessible to students from various disciplines (31). As with traditional UREs, graduate students also feel underprepared to mentor many students with little to no provided guidance (39). These challenges present considerable barriers for starting a robust CURE and discourage widespread implementation.

C. A hybrid approach to undergraduate research

To address the challenges present in both apprenticeship UREs and CUREs, several programs use a hybrid model (7, 40, 41). These research environments use the pedagogic tools associated with CUREs to teach students research skills, while retaining the open-ended nature of apprenticeship UREs. The use of familiar pedagogic tools can ease students into the real-world research environment. The structured framework also allows more opportunities for students to participate with less direct PI oversight, while better using graduate teaching assistance and peer-to-peer mentoring to promote collaboration further (42).

Building on this knowledge and in response to the limitations of traditional UREs and CUREs, we have created a hybrid framework for UREs called the “adaptive undergraduate research training and experience” (AURTE). This framework has been piloted in our group, the Brown Experiential Learning (BEL) program, focusing on computational biophysics research. The AURTE framework uses asynchronous training paired with direct-tiered mentoring delivered by peers, graduate students, and faculty and puts heavy emphasis on workforce development. By combining traditional pedagogic tools (syllabi, graded assignments, and training modules) with novel research and tiered mentorship, AURTE successfully promotes positive short-term outcomes in students. Our previous study showed the efficacy of a prototype of AURTE in teaching research skills and increasing student participation in BEL from the years 2010 to 2015 (43). Over that same time, BEL also saw an increased number of student coauthors on papers and presenters at research conferences. After over 9 years of active development and refinement of AURTE, we are now equipped to perform longitudinal studies to assess the framework’s efficacy in teaching lasting research skills.

This study intends to probe the long-term effectiveness of the AURTE framework, its impact on alumni and current students, and its efficacy in teaching enduring research skills and computational-based biophysical methods. The longitudinal impacts of this adaptive URE on student outcomes are analyzed by soliciting feedback from current and previous students. We also examine retention in our group compared with all broad metrics to evaluate the engagement of our program and the ability to keep students through their academic careers. By evaluating our group in this way, we can assess the quality of skills learned and our ability to scale our mentorship capacity broadly. Through our evaluation, we expect to generate new insights into the long-term efficacy of hybrid research environments and intentional mentoring in promoting positive student outcomes both academically and beyond graduation. We also expect to gain valuable insight into the effectiveness of our biophysical educational programs in teaching lasting, career-oriented biophysical research skills.

Here, we share insights into how the AURTE framework promotes positive mentor–mentee experiences for graduate and undergraduate students. We also discuss how others could implement the AURTE framework into UREs. This knowledge can inform the iterative improvement of AURTE and the design of future UREs at R1s. We expect AURTE to be useful for primarily undergraduate institutions because our training content can easily be adapted to a more CURE-focused curriculum that is generally found in primarily undergraduate institutions.

A. Continuation

Several components of AURTE have remained consistent in the BEL program since 2016 (43). Providing shared group expectations, a syllabus, weekly lab meetings, and the semester’s final deliverables (final paper and presentation) have continued to be successful and effective in delivering content to students and evaluating performance (43). However, several aspects of the AURTE framework have been expanded or added to ensure more sustainable outputs and increase scalability without sacrificing student learning or flexibility.

B. Training semester and module development

Key to the AURTE framework is the inclusion of holistic training, covering both general theory and computational biophysics specific methodology. Previously, students in the BEL program were trained primarily in the methodology needed for the assigned research project. Having highly specialized students created difficulties when trying to promote peer-to-peer assistance. Also, the responsibility of training undergraduate students was split between the PI, a graduate student, and the lab manager, leading to wildly varying rigor, expectations, and timelines for first-semester students and resulting in inequitable training experiences. To solve these issues, students in the BEL program now spend the entire first semester training and performing a microresearch project in an environment that uses many elements of CUREs, including graded assignments and predefined modules (Table 1). After completing the training semester, subsequent semesters remove some of the strict guidance, and students begin working with a graduate student on a research project. Including a training semester allows students to learn new skills in an environment in which mistakes are expected, without the pressure of working against critical deadlines. Many students complete the modules faster than others; however, the flexibility the modules provide helps enable success in students from various disciplines with varying levels of background familiarity. The training semester also provides students with a broad knowledge base of computational biophysical techniques. With a broad set of biophysical methodologies, students exit training more equipped to help each other and consult with outside collaborators for computational needs more efficiently.

Table 1.First semester in the Brown Experiential Learning program compared with continued semesters.

View Fullscreen

Canvas, the primary learning management system used at Virginia Tech (Blacksburg, VA) hosts our online training. Several modules have been developed as part of a consistent first-semester training experience. Previous studies show that modular learning content effectively trained students in protein–structure fundamentals (31, 36). The BEL program has developed modules to train students in various research skills and biophysical techniques. The modules were designed to be online and self-paced to ensure that students could learn asynchronously. The computational nature of our lab allows us greater flexibility with asynchronous learning and has enabled many students to complete assignments in a way most conducive to a preferred learning style. The asynchronous working environment proved invaluable during the pandemic, because the modules could be completed fully remotely and allowed the program to maintain operation and recruitment, despite being fully remote. The modules are split into 2 broad categories, advanced research skills (ARS) modules and computational biophysics modules. Virginia Tech University Libraries have developed the ARS modules and provide domain-agnostic research skills (44; Table 2). These provide all students with a fundamental foundation of research skills and have been heavily used in other experiential learning groups and first-year experience courses. Students also receive digital credentials by completing the ARS modules, helping students in first-year experience courses receive URE opportunities outside of BEL (Table 2).

Table 2.Overview of advanced research skills modules.

View Fullscreen

The BEL program modules are specifically designed to teach fundamental biophysical knowledge, data and computer literacy, and computational biophysics methodology (Table 3). These modules were designed to be cohesive, with each one building off the assignment of the previous one, using the same reference protein and structure. The modules allow students to practice individual skills, while obtaining a complete picture of how these various methodologies can be used in tandem to study the biophysical interactions of proteins (45). Throughout the past 5 years, since the start of this module-based training, the modules have been continually updated and expanded to ensure they are discussing the latest developments and tools used for the respective methodologies. The BEL program has also developed many modules to teach broadly applicable data science, programming, and data visualization techniques.

Table 3.Overview of the Brown Experiential Learning program modules.

View Fullscreen

The training semester provided benefits to the graduate student mentors. Shared and clearly outlined training and expectations meant that graduate students could spend less resources developing and planning individualized training. Broad biophysical training also allows graduate students to identify the strengths of each student and promotes informed discussions to identify an appropriate project. The modules continue to be adapted to teach biophysical methods in several outreach events for various audiences, ranging from high school students to graduate students from wet labs.

Public-facing releases of our module content are currently posted, and updated versions will be released on our Open Science Framework web page (https://osf.io/82n73/). Students take varying amounts of time to complete the modules, depending on the time devoted to training and prior skill sets. However, we maintain the required benchmarks for module completion. If a student works 6 h per week, modules 1–3 can typically be completed within 3 weeks, and modules 4–6 can be completed in 5 weeks. Additional modules are then assigned, depending on the project and interests of the student.

C. Transitions to a cloud-based framework

A major evolution of the BEL program was the transition to predominantly cloud-based tools and communication. The lab manager has overseen the transfer to cloud-based documentation and communication (43). The transition includes moving to digital lab notebooks via Microsoft OneNote, regular communications done via Microsoft Teams, and shared file networks for almost all research outputs. These tools allow for consistent version history, document restoration, and the ability to quickly disseminate information to a large team of graduate students and collaborators when working simultaneously on projects. Particularly when working with large computational simulation datasets, having consistent, shared file access allows for files to be quickly accessed and removes barriers to performing analysis. This cloud-based framework proved instrumental during the coronavirus disease 2019 (COVID-19) pandemic. During the pandemic, we maintained full-scale operations and increased student intake.

The lab manager also facilitated the expansion of the BEL program to expand into data science collaboration projects throughout the campus. The focus on data science collaborations allows students the opportunity to refine data management, analysis, and visualization techniques, which are becoming increasingly important in all facets of academia and industry. Students can focus solely on the data science aspects of the program, working primarily on collaboration projects.

When transitioning to cloud-based communication tools, there are several factors to remember. Regardless of the communication platform (Microsoft Teams, Slack, Discord, etc.), ensuring that all students are appropriately onboarded is critical to ensuring mass adoption and use of the tools. Consistent and frequent check-ins on these platforms (~2 per week) help students become accustomed to regularly reviewing these platforms for communications. Concern was also placed on ensuring that the platforms used were accessible without additional charge to the lab or the student, including after the student graduated. Also, understanding the specifics of the volume of data that students are collecting can help in choosing a file-sharing service that is best for your lab (Google Drive, Microsoft SharePoint, GitHub, etc.).

D. Remote and hybrid working

Remote working environments increase opportunities for student engagement and participation in biophysics research labs (13). The transition to cloud-based tools has allowed us to offer hybrid working environments for students in the BEL program. Increased flexibility has allowed more students, who would otherwise have limitations in finding or participating in research experiences, to join our lab. The flexibility was facilitated by having a remote desktop connection to a powerful Windows computer. Remote access to powerful hardware lets students access computationally expensive analysis programs without being physically present in the lab or personally owning powerful computers. Remote working options also allow us to work with students who do not have summer housing on campus and underrepresented populations who want to volunteer in our lab. Although we offer both in-person and online options, some students have elected to work entirely remotely.

A. Assessment

A survey was created to evaluate the long-term effectiveness of BEL in teaching lasting research skills (Supplemental Material Fig S1). The survey was designed to evaluate the effectiveness of BEL in imparting these skills during the students’ time in the program. The questions were created to evaluate both the engagement in the program on academics and its efficacy in imparting impactful career skills. The survey was sent out to BEL alumni, as well as current graduate and undergraduate students, for a sample population of 111. All questions can be found in the Supplemental Material. Surveys were distributed by an email list of all current and former students. In total, 27 responses were collected.

In addition to self-reported metrics on student outcomes, internal metrics regarding student enrollment and retention were compared with university-provided metrics of student enrollment and retention in undergraduate research. This work was carried out in accordance with the Virginia Tech Institutional Review Board’s standards and practices (protocol IRB 23-984). All university metrics were collected by using the Virginia Tech University DataCommons open access datasets (46). Visualizations were created by using Tableau (version 2023.3, Salesforce).

A. The AURTE framework retains students longer than the institutional standard

Compared with university retention statistics, BEL was found to have a much greater retention of students on a per-semester basis than at a university level (Fig 1). All data presented here do not count for the training semester: 60% of students who participated in a semester of research for university credit after the training semester stayed for a second semester, compared with 35% of students who participated in a second semester of research for credit at the university level (Table 4). The data indicate both the high retention of students our program encourages and how early we can support students working with us in research. The training semester allows us to take transfer students who may have received differing curricula at a previous institution. Internal metrics show that our training semester can support students earlier in the academic career, with many of our student researchers being lower-division undergraduates (Fig 2). Every year, our framework has more academic sophomores working in the lab than most labs at the university. We also have at least one first-year student each year, typically remaining in our lab for the entirety of their academic career.

View Figure

• Download as Powerpoint
• Download Figure

Table 4.Proportion of student semesters of continuous undergraduate research.

View Fullscreen

View Figure

• Download as Powerpoint
• Download Figure

Student retention is critical to a successful URE. Many studies have demonstrated that the greatest gains in both scientific understanding and curiosity are found in students who have taken ≥3 semesters of dedicated research experience (48, 49). Higher retention also confers greater career awareness and confidence (49). Although all UREs are valuable, maintaining direct mentorship over multiple semesters and having students transition from mentee to mentor over ≥2 academic years provide more career readiness than a single semester of training (50). By strongly emphasizing cultivating a collaborative working environment, developing robust computational biophysics training, and encouraging intentional tiered mentoring, we created an environment in which far more students can enjoy the benefits of continued undergraduate research. These benefits culminate in solidifying workforce-ready skills and greatly aiding students in scientific careers.

We also report the number of students who performed undergraduate research in our program each semester since 2017, not including students participating in the training semester (Supplemental Material Fig S2 and Fig 2). We have an average of 15 active, trained undergraduate student researchers each semester, demonstrating the persistent scalability of the program. These metrics also highlight how cloud-based communication and file storage allowed us to maintain our numbers and continue to expand throughout the COVID-19 pandemic from the fall of 2020 onward.

B. The AURTE framework conferred lasting research skills

Survey responses indicate that most alumni and current students found the pedagogy tools to be helpful in being successful in undergraduate research (Table 5). Participants indicated that the graded final project was the most universally helpful pedagogic tool used by the BEL program. These results speak to how feedback, from peers and mentors, is critical for accomplishing research goals and growing research skills. The helpfulness of the graded final project also highlights the importance of applying the skills they learned into a cohesive product and building a narrative that goes beyond the data. These results are bolstered by regular semester reflections at the start and end of each semester, which help the students self-identify roadblocks to research and personal areas for improvement. The formal training semester also appears largely successful in helping most students be successful in undergraduate research, with many students expressing the value of learning a wide range of topics in biophysics, rather than only the methodologies required for the research project. These results are consistent with recent studies using modular-based content to teach biophysical methods and concepts (31, 36).

Table 5.Survey responses to the question “Which of the following helped you be successful in undergraduate research?” (n = 27).

View Fullscreen

The BEL participants indicated that BEL positively affected several aspects of academic learning and personal durable skills (Fig 3). Every student indicated that BEL had a positive or somewhat positive effect in increasing confidence in the ability to perform research, honing a skill set in the intended career path, and presenting scientific findings in an oral and written format. The positive effect on all learning outcomes indicates that holistic teaching on fundamental biophysical knowledge and a structured environment for learning the methodologies provide positive outcomes for student understanding of biophysical methods and for durable research skills.

View Figure

• Download as Powerpoint
• Download Figure

C. Mentors and mentees benefited from tiered mentorship

Participants who indicated being mentors as part of BEL (n = 7) were asked follow-up questions regarding experiences. Students indicated that mentoring increased confidence and understanding of core research skills. Five of the 7 mentors indicated that mentoring extensively enhanced communication skills because they had to learn to explain scientific concepts and personal expectations to the mentees. Four mentors mentioned the value of a better understanding of concepts through teaching. Mentors also cited having improvements for troubleshooting and their confidence as scientists because of mentoring (Tables 6 and 7).

Table 6.Responses to “How did mentoring students affect your understanding of research skills?”.

View Fullscreen

Table 7.Responses to “What did you learn while mentoring students?”.

View Fullscreen

Mentoring is a critical component of undergraduate research. The effectiveness and amount of mentoring students receive are directly correlated to student retention in STEM majors and the decision to become researchers (9). By incorporating a tiered mentorship approach, while promoting peer-to-peer interaction, BEL has created a scalable foundation to use known hierarchic mentorship best practices (51). Our system allows graduate students to practice mentorship skills and provides a pathway for long-term undergraduate students to become more involved with mentorship. Of the 7 mentors surveyed, all 7 were undergraduate students who performed research with BEL, with 3 having continued to work in the lab as graduate students. The elevation of undergraduates from mentee to mentor allows students to practice a new set of communication and project management skills, while also deepening scientific knowledge.

The key to a tiered mentorship program is ensuring adequate mentorship training for hybrid mentor–mentee roles. Although the many benefits of quality mentoring are known, mentoring without training can lead to frustration for both the mentor and mentee (8, 34, 52). The success of our program hinges on slowly introducing students who want to mentor into the role, giving them frequent advice and helping them establish a mentorship philosophy. Although there is not a one-size-fits-all solution to mentoring every student, we can empower our graduate student mentors to engage in mentoring actively. These intentional mentorship strategies create a beneficial, sustainable mentorship experience that engages undergraduate researchers and promotes graduate student identities as scientists.

D. Alumni used skills taught in the AURTE framework

Alumni (n = 14) were asked additional questions regarding the retention of skills learned in the AURTE framework and which skills they used since graduating. Nearly all alumni reported heavily using time management skills, with data visualization, management, and analysis also being heavily used (Table 8).

Table 8.Survey responses to the question “Which skills have you utilized since graduation?” (n = 14).

View Fullscreen

Alumni indicated that the skills learned in BEL were largely applicable to their careers. Although the broader skills were more applicable, knowledge of molecular modeling and bioinformatics was still useful to over a third of alumni since graduating. These responses align with responses to postgraduation career paths for our students (Table 9).

Table 9.Survey responses to the question “What have you done post-graduation?” compared with respondents’ majors (n = 14).

View Fullscreen

E. Challenges and future considerations

One potential limiting factor of building a large research lab is ensuring that the lab has continued support and management from the faculty, staff, and graduate students. Management of a larger lab has to be intentional and requires the collective effort of all involved. One challenge associated with larger labs is the struggle to build community. Although the self-paced learning modules have increased flexibility and decreased the resources needed for students to achieve a broad understanding of our methodologies, building community has been more challenging than traditional URE experiences. The program has had to offer community events intentionally, listen to the students’ desires, and foster mentor–mentee connections. Starting in the spring of 2024, we now require first-semester students to spend half of the time in person working in the lab, and in subsequent semesters, give them the flexibility to work in an environment where they can succeed. We heavily emphasize allowing our students to explore where they can be successful without prescribing a particular strategy or environment. We have observed in our students, graduate students, and faculty a wide range of preferences and readiness to work remotely or in the lab and emphasize that both can be successful paths, so long as both are explored.

The other challenge associated with creating a robust training semester has been balancing providing a large library of training materials, while continuing to promote scientific curiosity and independence in students. Although the CURE elements of the training semester provide a more familiar environment for students to learn complex methodology, many enter the second semester of BEL expecting research to be closer to doing standard coursework than the realities of a research environment. To ease this transition, the structure of lab meetings in the spring of the 2024 semester moved away from lecture-based meetings focused on teaching methodology to graduate student seminars focused on current research and increasing student participation. Although methodology-based lab meetings were essential to training larger cohorts of undergraduates before developing and refining the learning modules, we believe the modules, combined with careful mentorship, are sufficient to teach new cohorts the methodology. We believe transitioning to more traditional lab meetings will enhance students’ intrinsic motivation, while simultaneously connecting learning techniques to current research topics sooner.

Overall, the BEL program and AURTE framework will need to be flexible to balance the many competing interests, while remaining structured to facilitate growth. The key to remaining flexible is keeping communication open and allowing problems to be communicated to the group without fear or shame. Monthly leadership meetings are being created among the faculty, staff, and graduate students to facilitate communication. These monthly leadership meetings will provide an open forum for continued improvement, ensure that all voices are heard, and allow for all needs to be met.

A. Takeaways and implementation strategies

Based on our findings, we found several factors that contributed to the success and scale of our group that others can use in undergraduate research training programs:

1. Robust, intentional, systematic training ensures that all students reach a baseline understanding of biophysical theory and techniques. Structuring the education and grading in undergraduate research allows students to focus on learning, while becoming comfortable with the unknowns inherent to research. By outlining clear goals and metrics for success as it pertains to undergraduate student academics, more time can be spent with a mentor exploring fundamental research skills and concepts. Implementers of this framework should seek to build or use proven pedagogic techniques similar to those used for CURES. Build an education suite that allows for flexibility and variability among students. Assume a student comes in with very little prerequisite knowledge. Those students with an advanced background in the domain will quickly go through the content, while still allowing those students who need that additional information the time to learn. Our training materials and other tutorials can be found on our Open Science Framework (https://osf.io/82n73/).

2. Use cloud-based resources to facilitate regular, authentic, productive communication and data management. Ensure that all students know and understand communication expectations, while providing flexibility and openness for students to contact mentors for advice or clear roadblocks. Implementers should consider that communication styles and needs will change for the students as time progresses. AURTE emphasizes adaptability as the students change, and communication will change with them as the students grow, experience environment changes, and encounter new or unforeseen challenges.

3. Both mentors and mentees should always strive for continuous learning. There is no point when undergraduate mentees or graduate student mentors are “finished” learning. Present clear pathways for undergraduate students to become mentors and maintain the expectation that graduate students will mentor undergraduates. Although organic mentor–mentee relationships are desirable, explicitly establishing a mentor–mentee dynamic allows for shared expectations and clear paths to growth.

Future consideration will involve further standardizing AURTE to be broadly applicable to other research groups and expanding these results to measure the efficacy of the structure in wet lab environments. Undergraduate research is foundational to training the next generation of scientists to tackle the problems of the world. It is our duty as the current generation of researchers to strengthen that foundation by iterating and evaluating URE structures. By expanding opportunities for research, we hope to empower every future researcher with the tools and techniques to succeed. Through continued development of AURTE, we seek to facilitate expanding opportunities for undergraduate research both in and beyond biophysics.