A ChatGPT-Assisted Reading Protocol for Undergraduate Research Students
A major challenge for undergraduate students is reading scientific literature. This is especially true in biophysics, where many of the concepts may not have been covered in undergraduate courses. Students can become overwhelmed, which may lead to less overall engagement with the scientific literature. In response, we have developed a guided reading protocol that combines prereading strategies, structured note-taking, and ChatGPT to help students clarify unfamiliar concepts in an interactive way. To test the protocol, participants in this study were given an initial survey to determine their experience with reading scientific literature. After this, they were given an abridged biophysics paper and the protocol. The ChatGPT transcripts were analyzed by using open coding, and the students were given a poststudy survey. We found that most students did not appear to regularly engage with the literature, possibly because of content barriers they encountered. Analyzing their transcripts, we observed that students asked for definitions, explanations, summaries, and simplifications. Overall, students reported that using ChatGPT was a positive experience and that they expected to use ChatGPT in the future. From this work, we expect that this new protocol may be a way to keep novice students from becoming discouraged when reading scientific papers and keep them engaged with the current literature.ABSTRACT
I. INTRODUCTION
When a student joins a research laboratory, they encounter scientific literature, often for the first time, that goes beyond what they have encountered in their previous coursework. Undergraduates engaged in biophysical research, especially, must grapple with structural biology and biophysics literature with unfamiliar terms, concepts, and methodologies. Encountering these new ideas and the complexity and unique format of scientific works acts as a barrier, preventing students from engaging with the literature in a meaningful way. Without support and guidance, reading this literature may cause anxiety that can hinder the learning experience for the undergraduate researcher (1). Gaining competency in reading the literature is necessary not only for developing scientific skills but also contributes to improving student motivation and sense of belonging in their field, which is critical for retaining students in the sciences (2–6).
Although this is a valuable topic for training students to be scientists, limited studies on how undergraduate science, technology, engineering, and mathematics (STEM) students approach reading scientific literature are available. A study by Lennox and colleagues indicated that undergraduates use multiple approaches to finding evidence when reading primary scientific literature, including prioritizing different sections of the paper and using figures and text to support conclusions; however, they also found that these students sometimes skim or even skip numerical data presented in the text (7). Although most students were able to identify the main points of a given text, they struggled to provide evidence for their conclusions. A study by Mitra and colleagues observed that many undergraduate STEM students struggled to identify the research hypothesis of articles while participating in a guided literature-reading curriculum (8). Additionally, studies observed a disconnect between students’ self-perception of their scientific reading comprehension ability and their actual skills, which may contribute to undergraduate students not fully engaging with literature because they do not realize that they are not doing so effectively (7, 8). Reading scientific literature requires different skills from casual reading, including prioritizing which section to read, understanding the context of quantitative data or figures with relation to the text, and knowing how to decipher technical terms and jargon (6). As a result, skills for understanding scientific literature should be explicitly taught to undergraduate research students along with the use of new technologies, such as ChatGPT, to help students fully engage with the material.
A. Scientific and pedagogic background
Interventions have been developed and assessed to help undergraduate STEM students engage with scientific literature more effectively. In a project by Round and Campbell, a template called “figure facts” was used to help students read and interpret scientific literature by having students focus on experimental data presented in an article’s figures. Students showed improvement in data interpretation and attitudes about reading scientific literature (2).
Mitra and Wagner developed a curriculum to guide undergraduates’ reading of scientific articles by focusing on the hypothesis and conclusions drawn from experimental results and using a series of guided worksheets and workshops; they found that students’ confidence improved with their curriculum (8). Gold and McNeal embedded a scientific literature–reading and graph-interpretation curriculum into a 10-week research experiences for undergraduates (REU) program (6). They analyzed the reading strategies used via survey and eye-tracking studies to compare REU participants to experts and found that with the embedded curriculum and authentic research experience in the REU, participants made gains in their reading strategies and graph interpretation skills (6). To advance undergraduates’ scientific literacy and understanding of the publication process, Otto and colleagues developed a curriculum in which students peer-reviewed preprints, which was found to improve students’ scientific literacy and sense of belonging in STEM (9).
These interventions have greatly helped students’ understanding of the format and reading of scientific literature; however, conventional methods are unable to address the issue of encountering unfamiliar terms within the context of a research article in a timely manner to not frustrate and discourage the student. This is especially critical when undergraduates are conducting research and reading journal articles in a multidisciplinary field such as biophysics, where they need to engage with terms that are outside the scope of the material they have learned in their classes. This study shows that a guided reading protocol that includes artificial intelligence (AI)–based large language models (LLMs; e.g., ChatGPT) can be used to bridge knowledge gaps and help undergraduate research students effectively read and understand scientific literature.
The advent of LLMs such as ChatGPT has quickly changed how individuals interact with information. These tools can be used for answering questions and summarizing information and can play a role in improving equity by minimizing language barriers for diverse learners (10–13). Many remarkable use cases for AI in postsecondary science education are being explored and analyzed in the literature. A common theme in published studies is the value of training students in AI literacy so that they understand how to use a tool such as ChatGPT effectively and understand its limits (4, 10, 14–18). Further, much of the literature has highlighted the need to develop training for students in prompt engineering, the process of crafting AI queries to elicit better answers (13–17).
Tassoti developed a structured framework for posing questions to ChatGPT in a way that led to satisfactory responses without the need to copy large sections of text into the chat (16). Although nearly all studies stress the significance of prompt engineering, studies on the use of LLMs in STEM education vary in how the students interact with the chatbot. In one study by Exintaris and colleagues, students were introduced to problem-solving methods, taught a metacognitive scaffolding approach for problem solving, and then asked to critique a ChatGPT-generated solution to a problem (19). Most students were able to find flaws in the LLM-generated results, which shows that with scaffolding, students were able to think critically about the responses provided by ChatGPT (19).
A study by Pence and colleagues involved analyzing a three-part assignment for undergraduate chemistry students. Students used ChatGPT to summarize an article from a chemistry trade magazine, analyzed ChatGPT output for a numerical solution, and used ChatGPT to analyze primary scientific literature by using it to answer specific questions (20). In summarizing the trade magazine article, the students largely believed that this was a good use of the LLM and that the summaries were generally accurate but superficial (20). For the assignment that involved ChatGPT to answer questions from primary scientific literature, results were mixed. Students became frustrated with prompting; some hallucinations occurred, and some superficial answers were offered. However, when analyzed by faculty, it was found that most responses from ChatGPT were at least partially correct (20).
To facilitate effective engagement with scientific literature for undergraduate research students, we have developed a new approach that combines prereading strategies adapted from McGuire’s book, Teach Students to Learn; note-taking methods; and the use of AI-based LLMs, such as ChatGPT (21). The LLM assists students in clarifying unfamiliar terms and concepts, preventing these from becoming barriers and thus enabling students to better integrate the paper’s content into their own research.
II. METHODS
Research subjects were recruited via email from a pool of undergraduate students participating in research in the chemistry department at a primarily undergraduate institution. The students ranged from second- to fourth-year students with <1–2 years of undergraduate research experience who had majors in chemistry, biochemistry, and biology. A pre-intervention survey was given to assess students’ prior experience and comfort level with reading scientific literature and with using AI chatbots such as ChatGPT.
After completing the initial survey, students were sent two items: a ChatGPT-assisted literature-reading protocol and the title, authors, abstract, and introduction of the 2018 paper “Stabilized Coronavirus Spikes are Resistant to Conformational Changes Induced by Receptor Recognition or Proteolysis” by Kirchdoerfer et al. (22). This structural biology paper was chosen because we expected it to present unknown terms and methodologies to chemistry or biology research students but also provide some familiarity with the coronavirus subject matter. The paper was abridged because the introduction is often read first by students, and it is where many unfamiliar terms and concepts that may act as barriers would be presented.
The ChatGPT-assisted literature-reading protocol comprised four elements to guide the students in using the chatbot as a tool for understanding scientific literature. This procedure is also included in the Supplemental Material as Supplemental Figure S1.
Preview of the paper: analyze the title, abstract, headings, and figures to gain meaning
Title: identify key words that provide the paper’s main focus
Abstract: analyze key findings and the general format of the abstract (i.e., significance, problem to be addressed, approach, and insights)
Headings and figures: skim text headings and figures to anticipate the questions the paper will answer (note that the abridged paper we provided did not include a figure)
Guided reading of the paper: highlight key text with an erasable method and refine or edit highlighting as they read; make margin notes of significant points and, if needed, extended notes in a separate notebook
Use of ChatGPT for clarifications: use ChatGPT to help clarify unfamiliar concepts, terms, or methodologies that are stumbling blocks; students learn how to effectively use ChatGPT for this purpose:
Introduction: the reader introduces themselves with their academic and research background and any other pertinent information
Paper introduction: the reader informs ChatGPT about the paper, providing the paper’s title and general field
Questions: the reader asks questions to ChatGPT and is encouraged to ask clarifying or follow-up questions as needed
Post-reading summary: the reader should review their notes and ChatGPT transcript to write a summary of the paper; it can be as brief (i.e., one or two sentences, bullet points) to synthesize ideas and serve as a record of the paper for the reader.
This protocol was designed to be a general procedure for reading papers, and as a result it includes instructions in the previewing step that directs students to analyze headings and figures. This was an initial pilot study, and we focused on the introduction of the provided scientific paper, which did not include figures. Our goal was to explore generally how students interacted with the chatbot in this first study. In future work, we will test the protocol on an entire paper, which will provide insight into how students interact with figures in the paper.
Students were asked to save their ChatGPT transcripts as PDF files and send them to researchers via email upon completion of the guided reading task. The ChatGPT transcripts were anonymized before analysis. Students were then sent a post-implementation survey about their experience using the ChatGPT-assisted literature-reading protocol. We did not ask students which version of ChatGPT they used; however, we do not believe this had an effect on this study. The fundamental functions needed for this protocol—interactivity, summarizing, and explaining—are available across all versions of the chatbot.
Responses to multiple-selection and Likert-scale questions on the pre- and post-implementation surveys were tallied, and free response questions were coded by using open coding for common themes (23). The ChatGPT transcripts were analyzed in two ways. First, the number of questions asked by each student was tallied, and second, open coding was used to analyze the types of questions being asked to ChatGPT. Five question-type codes emerged. Questions corresponding to these codes were tallied.
III. RESULTS AND DISCUSSION
A. Pre-implementation survey
In the pre-implementation survey, students were asked about their experience with and perceptions about scientific literature. The survey questions that students were asked, along with their responses to the survey, are included in Supplemental Material 1. Fourteen students responded to this survey. The first portion of the survey asked about experience reading journal articles, and the second part of the survey asked about approaches to reading journal articles and previous experience using LLMs such as ChatGPT.
The first set of questions focused on understanding students’ experience and comfort with reading scientific papers (Fig 1). These students were asked whether their research advisors or classes taught them how to read papers, how confident they felt about reading papers, and how often they actually read papers. Students reported mixed views on how well they believed their research mentors and course instructors prepared them to read scientific literature, with 43% (6 of 14) of students agreeing or strongly agreeing that research faculty had taught them to read scientific literature, whereas 50% (7 of 14) felt the same about their courses.
Citation: The Biophysicist 6, 1; 10.35459/tbp.2024.000281
Students were also asked to rank their confidence in reading scientific literature, with 57% (8 of 14) of respondents saying they were confident or very confident and 29% (4 of 14) saying they were moderately confident reading literature. However, this self-reported measure did not correspond with students’ actual limited experience reading literature, which is in agreement with previous studies showing that undergraduate students often overestimate their reading skills (7, 8). Only 43% (6 of 14) of students agreed or strongly agreed that they read papers regularly. When asked how many journal articles students read per month related to their research project, 29% (4 of 14) of students reported not reading any journal articles, 64% (9 of 14) reported reading one to two articles, and only 7% (1 of 14) reported reading three to four articles (Fig 2). When asked how many articles they read per month in their classes, 57% (8 of 14) of students reported not reading any, 21% (3 of 14) reported reading one to two articles, 7% (1 of 14) reported reading three to four articles, and 14% (2 of 14) reported reading more than four articles.
Citation: The Biophysicist 6, 1; 10.35459/tbp.2024.000281
When asked about approaches students take to reading scientific literature, allowing for multiple responses, the most common approach was reading the abstract first (64%, 9 of 14) followed by reading the article from beginning to end (50%, 7 of 14), and finally jumping around the article (50%, 7 of 14; Fig 3). Interestingly, no students responded that they read the conclusions first. Students were also asked to provide free responses to describe reading strategies they have been taught. The most common strategy students reported was to read the abstract first and then look at the figures (36%, 4 of 11).
Citation: The Biophysicist 6, 1; 10.35459/tbp.2024.000281
We also asked students what they do when they encounter an unfamiliar word, phrase, or concept, allowing for multiple responses (Fig 4). The most common response was using a search engine, such as Google (93%, 13 of 14), followed by using LLMs, such as ChatGPT (50%, 7 of 14). We can see that undergraduate student researchers are using online resources to decode unfamiliar terms and concepts but generally without specific guidance that would effectively and efficiently help them.
Citation: The Biophysicist 6, 1; 10.35459/tbp.2024.000281
In a write-in response question, the students were asked what they perceive to be the most difficult aspect of reading scientific literature. Two common themes emerged: encountering unfamiliar vocabulary, terminology, and abbreviations (46%, 6 of 13) and deciphering scientific writing in terms of its length, density, and style (54%, 7 of 13). Although the undergraduate researchers in this study expressed confidence in their ability to read scientific literature, they also indicated a lack of experience and instruction on how to approach reading journal articles.
Undergraduate students appear to be using tools such as search engines and to tackle the challenges of reading scientific literature but still perceive significant barriers to their understanding. This suggests that training research students to use LLMs such as ChatGPT can be a productive tool for removing the obstacles of unfamiliar terminology. It may also be beneficial for undergraduates who are tackling difficult scientific literature, possibly for the first time.
B. ChatGPT transcripts
A total of 11 students completed the ChatGPT-assisted reading of the article and returned the ChatGPT transcripts for analysis. Anonymized ChatGPT transcripts were analyzed for number of interactions initiated by each student and common types of questions and statements that were posed by the students. Here, we counted interactions as either giving background information to ChatGPT or asking a specific question. The average number of interactions with ChatGPT was 7.1 interactions, with a range of 5 to 14 interactions in each transcript.
To investigate how undergraduate research students interacted with ChatGPT, open coding was used to group the types of questions and statements posed into five common themes: a description of the student’s educational background, definitions of unfamiliar terms, requests for an explanation, questions to the chatbot asking to summarize text or the conversation, and simplifications of the material. The number of instances of each of these themes is summarized in Figure 5 and further discussed here.
Citation: The Biophysicist 6, 1; 10.35459/tbp.2024.000281
1. Background
Students provided information about their educational background similar to the way they were prompted in the user guide (82%, 9 of 11 students used this type of interaction). When providing background information, students offered information about their educational background, and some gave information about their experience reading scientific journal articles. Students were prompted to present information about their academic background in the provided reading guide to allow ChatGPT to tailor its response to the individual’s educational level.
All students who provided background information described their educational background in terms of major and relevant courses completed. For example:
I’m currently a third-year chemistry student in my second semester. I’ve taken two semesters of general chemistry and organic chemistry, each with analytical chemistry and quantum mechanics classes. I am enrolled in a thermodynamics class right now. I have a very light background in biology, mostly pertaining to protein structures, but chemistry is more of my strong suit.
Of the students who gave background information, 44% (4 of 9) also gave information about their research interests and experience, which was in addition to what they were prompted to provide in the reading guide. For example:
I am a third-year college student majoring in biology. I have taken courses such as general chemistry, organic chemistry, and biology courses such as microbiology and now biochemistry. I also currently do research in a chemistry lab doing analytical chemistry comprised of measuring absorbance using a UV-Vis spectrometer. I rarely read scientific papers, and it takes me a long time to thoroughly comprehend even a few sentences.
2. Definitions
Students asked for definitions of unfamiliar terms (100%, 11 of 11 students used this type of interaction). All students asked definition questions pertaining to unfamiliar words and phrases in the text. We saw in the pre-implementation survey that unfamiliar terms are common stumbling blocks in reading scientific literature. Using an LLM to obtain definitions quickly without needing to evaluate multiple sources may help undergraduate researchers to persevere in reading scientific literature quickly and help keep them from feeling discouraged when encountering new concepts. Example definition questions:
What does the word ‘zoonotic’ mean?
What is single-particle cryo-EM, and what does an ectodomain do?
3. Explanation
Students asked for explanations about concepts from the text (82%, 9 of 11 students used this type of interaction). In these questions, students asked for an explanation rather than just a definition about the text itself. Explanation questions frequently came after a definition question because the student wanted to further understand a new term or idea within the context of the definition they received. Examples include:
Why is it important to prevent the transition from pre-fusion to post-fusion states for the stabilized coronavirus spikes?
Why are coronavirus spike proteins trimeric? Does this contribute something important to their function?
4. Summarize
Students asked for a summary of information from the text or a summary of the ChatGPT response (36%, 4 of 11 students used this type of interaction). Summarize-type questions posed by students took two different forms. Most students gave either the digital object identifier (DOI) of the paper or large quotes from the paper and asked for a summary of the text. This may have been done to lower cognitive load for the reader when encountering scientific literature on an unfamiliar subject. An example:
The paper I need help understanding is ‘Stabilized coronavirus spikes are resistant to conformational changes induced by receptor recognition or proteolysis’ (DOI: 10.1038/s41598-018-34171-7). Could you please summarize the abstract?
Students also used summarize-type questions to give a summary of their understanding and prompt ChatGPT to respond and add clarification to the summary to critique and deepen their own understanding. An example:
I have just completed reading the abstract and introduction of ‘Stabilized coronavirus spikes are resistant to conformational changes induced by receptor recognition or proteolysis,’ a biochemistry journal article. The intro gives the background behind the experiment and its importance while the abstract summarizes the article in a brief paragraph. The intro discusses coronavirus strands like MERS-CoV and SARS-CoV but focuses on the latter. The virus contains large trimeric S-glycoproteins, which aid in recognizing host receptors and virla [sic] cell entry. As SARS-CoV breaks down into subunits S1 and S2, with S2 having the S2 cleavage site and S1 with N terminal domains and receptor binding domain. Using cryo-EM analysis, the researchers were able to develop to stop [sic] the transition of pre to post fusion states.
From this example, after the student shared their summary, ChatGPT provided definitions of many of the terms the student used (e.g., subunits S1 and S2) without prompting, and the student replied with follow-up definition and explanation questions. This interaction allowed the student to check and organize their understanding of the material and receive details and feedback from ChatGPT.
5. Simplify
Students asked ChatGPT to either simplify language from the journal article or simplify its own explanation in terms of language used or length of explanation (46%, 5 of 11 students used this type of interaction).
Students approached asking for simplifications in different ways. Some primed ChatGPT in their introductions to use simple language in its responses. These students may have recognized that complex language and unfamiliar terms are a barrier for their own understanding of scientific literature before reading the article. For example:
I am going to be using your help to analyze and understand a journal article about coronavirus. The article is titled ‘Stabilized coronavirus spikes are resistant to conformational changes induced by receptor recognition or proteolysis.’ When you explain things please keep it simple and understandable.
Other students requested simplification as they read different parts of the article to improve understanding of challenging material. For example:
Could you please simplify the third paragraph in the introduction?”
Or, after quoting the article:
Break this section of the paper down for me and explain this process in simpler terms.
In general, undergraduate researchers engaged with ChatGPT in various ways based on the ChatGPT-assisted reading protocol provided to them. Most students (82%, 9 of 11) followed the protocol’s recommendation of providing an introduction before asking questions to allow ChatGPT to provide appropriate-level responses. By far, the most common type of question asked was to provide a definition, with all students asking definition questions and a total of 31 definition questions being asked from all participants. This implies that a first barrier for undergraduate researchers reading literature is encountering unfamiliar words and concepts.
As discussed in Round and Campbell’s work, students reading scientific articles are likely to skim or skip unfamiliar or challenging terms, which prevents them from fully engaging with the literature (2). We observed that fewer students asked the chatbot to summarize or simplify the text, so it may be worthwhile to include explicit instructions and suggestions of different ways to interact with the chatbot in future versions of the protocol. This will be explored in future studies. By providing the students with a ChatGPT-assisted reading protocol, undergraduate researchers were given a useful way to obtain definitions for unfamiliar terms and to ask follow-up questions using their natural language. The undergraduate researchers could also ask ChatGPT to simplify language when needed, further lowering the barrier to engaging with the text.
C. Post-implementation survey
After completing the reading activity with the ChatGPT-assisted reading protocol, students were asked to complete a post-implementation survey about their experience using the reading protocol and their experience with ChatGPT. Eleven students completed the survey. The questions and responses are provided in Supplemental Material 2.
Students were asked whether they had previously used a reading protocol and whether they had used ChatGPT before (Fig 6). We found that 73% (8 of 11) of students had not used a reading protocol before participating in the study. From the survey, 91% (10 of 11) of the students had used ChatGPT before this study. When asked how they used ChatGPT before this study in a free response question, 50% (5 of 10) reported using it to understand concepts from class or research, and 20% (2 of 10) reported being required to use it for class assignments. Other students reported using ChatGPT for nonacademic related uses. This shows that most students had familiarity with ChatGPT before using the ChatGPT-assisted reading protocol in this study.
Citation: The Biophysicist 6, 1; 10.35459/tbp.2024.000281
Students were also asked whether they followed the procedure steps (Fig 7). Most students (82%, 9 of 11) followed the procedure exactly. The rest of the students either did not follow it at all (9%, 1 of 11) or followed some parts but not others (9%, 1 of 11). A free response was provided for students who did not follow the procedure. One student responded:
Citation: The Biophysicist 6, 1; 10.35459/tbp.2024.000281
I did not take any notes on the side, anything I had a question about was asked to ChatGPT.
This is an interesting response because it suggests that the use of ChatGPT might serve as a way to interactively take notes while reading the paper, which may be worth exploring in future work.
We also asked students whether they asked clarifying or follow-up questions after asking their initial question (Fig 8). None of the students said they asked follow-up or clarifying questions all or most of the time. We found that 27% (3 of 11) of students reported sometimes asking follow-up questions, along with 27% (3 of 11) who said they asked follow-up or clarification questions once or twice. We also found that 46% (5 of 11) said they never asked follow-up questions, which corresponds to the results we found in the open coding of student ChatGPT transcripts. Again, this suggests to us that we should include more explicit instructions in the reading protocol so that students can consider asking different types of questions when they need to understand a concept.
Citation: The Biophysicist 6, 1; 10.35459/tbp.2024.000281
The students were also asked to write in free response items on the biggest benefit and challenge of using ChatGPT while reading a scientific article. For the biggest benefit, the most common theme in response was that ChatGPT provides definitions and explanations in the context of the article (46%, 5 of 11), followed by clarity of explanation (36%, 4 of 11). Within their responses, students also expressed appreciation of the dynamic nature of interacting with ChatGPT (see the second and third quotes) as opposed to traditional search engines:
Having unknown words be defined in the context of the article.
The biggest benefit of using ChatGPT was not only being able to receive an answer, but also being able to tune how concise I wanted that answer to be. It’s also extremely helpful that I can ask a question in the context of a certain topic, and the computer will adjust accordingly.
The biggest benefit was having things broken down to me in a way that I could actually understand. I feel that when searching things up, it’s hard to find answers that actually explain without just reiterating what originally confused you.
When asked about the biggest challenge of using ChatGPT while reading scientific literature, the most common response was that it was a challenge to learn to phrase questions to get the desired response (46%, 5 of 11) and that the ChatGPT responses were sometimes too long (18%, 2 of 11):
The biggest challenge is figuring out how to ask ChatGPT what I want to know.
I feel like the biggest challenge using ChatGPT was providing enough background so that my question could be answered fully, coupled with the fact that I had to guess if I was providing enough context for a prompt.
Finally, students were asked in a free response question if they would use ChatGPT to assist reading scientific journal articles in the future. All respondents (100%, 11 of 11) stated that they would use ChatGPT to assist them in reading articles going forward. Common themes in responses were that ChatGPT provides context and explanations within the context of the article (46%, 5 of 11) and clarity of explanations and summaries (36%, 4 of 11):
I’d would [sic] expect to use ChatGPT in the future because it takes plenty of the guesswork out of contextualizing certain procedures and terms when reading papers on a research topic I am not familiar with.
I do expect to use ChatGPT in the future when reading scientific journal articles because it really does make the reading process much more easier, efficient, and effective. The program provides detailed, relevant, and real time explanations on questions and curiosities. It is also willing to word things differently and break down the information as much as necessary until you grasp the material. You can also provide it with your exact assumptions and thoughts on questions and material and it will respond to you and explain why you are right, wrong, or what it [sic] missing.
IV. CONCLUSIONS
Based on the feedback from undergraduate research students in this study, the ChatGPT-assisted reading protocol was seen as beneficial. Students predominantly asked ChatGPT definition-type questions while reading the introduction of the article during the study, and their feedback focused on the fact that unlike traditional search engines, ChatGPT provides definitions and explanations with respect to the context of the article. The students also valued the clarity and tunability of explanations provided by ChatGPT. The largest barrier to using ChatGPT was perceived to be formatting questions to elicit the desired response.
Especially interesting for this research was the finding that all surveyed students said they would use ChatGPT in future paper reading. The use of a chatbot while reading unfamiliar material may be a positive intervention to help students continue to engage with biophysics literature without becoming discouraged. If a tool such as ChatGPT can help students quickly find the information they need to continue reading a paper and continue asking questions until they are satisfied with their understanding, they may be encouraged to continue reading papers on a regular basis as they continue to develop expertise. Especially compared with using a search engine to find material, which can require a level of expertise to correctly frame the question and find the answer, the interactive, back-and-forth nature of the chatbot may make it easier for students to minimize disruptions to their reading and not feel overwhelmed when reading biophysics papers. Additionally, this type of approach may be useful in other contexts, such as helping students read textbooks or other material in their course work. In future investigations on using a ChatGPT-assisted reading protocol, additional training in effective question construction and different types of questions that can be asked would be beneficial to further remove barriers for undergraduate student researchers to effectively engage with literature. We also will be exploring how student interactions with the chatbot change when they read an entire journal article, with a focus on how they use the chatbot to understand quantitative data and figures.
Responses to questions about student experience and confidence reading scientific literature.
Student responses to how many papers they read per month for research or science classes.
Student survey responses on how they approach reading scientific journal articles; students were allowed to pick multiple responses.
Student responses to how they respond when encountering unknown concepts in scientific journal articles; approaches have been abridged for figure clarity. The full survey question can be found in Supplemental Material 1.
Number of students using the common interactions determined from open coding of student ChatGPT transcripts.
Survey of students’ previous experience with reading protocols and ChatGPT.
Student responses to whether they followed the reading procedure. The original question posed was, “When reading the abstract and introduction to the scientific journal article with ChatGPT, did you follow the procedure in the reading guide?”
Student responses to whether they asked follow-up or clarifying questions. The original question posed was, “When you used ChatGPT to help with unfamiliar words, phrases, or concepts, did you ask ChatGPT follow-up or clarifying questions after your initial question?”
Contributor Notes