You've sent a bunch of emails to potential mentors. A few have gotten back to you and told you about some projects they have available for you to work on. Some sound interesting, while others sound more complicated and time-consuming. How do you evaluate which project to take on? How can you tell which one's going to succeed and which ones will fail?

Throughout my medical training, I basically took whatever project I was offered. I didn't evaluate their potential or their likelihood of success. As a result, I spent years doing research that didn't lead to any publications. There are three key factors that decide a project's success - the project, the mentor, and your determination. The goal of this article is to help you choose the right research project so that your efforts culminate in a publication. I've written a long article about finding the right mentor, which can be downloaded after you join our newsletter.

So let's dive into the different things you should look at before agreeing to a research project. The foundation of any successful research project starts with the research question itself. Before you evaluate anything else about a potential project, you need to assess whether the fundamental question is worth pursuing.

The Research Question

Is the research question clearly defined?

I've previously written an in-depth article on how to ask a research question. But if the project you're about to take on does not have a clear research question, I would strongly consider finding a different project. This is basically a deal-breaker to me. One of the projects I wasted months on started out like this. The guidance was to collect the data, and after the data was collected we'd search for possible associations. At the time, I thought this was a good plan, but looking back, this should have been a red flag.

Will the findings of the research question be interesting whether the results are positive or negative?

After evaluating whether the research question is clearly defined, you should consider the findings that could possibly come out of this project. I would hesitate to commit to a project where the paper can only be published if the initial hypothesis was found to be positive. Lots of research questions can have negative results. In fact, the underlying assumption of all research questions is that there is no association between exposure and outcome (i.e., the null hypothesis). As such, the ideal project you pursue should be interesting enough to publish regardless of whether the results are positive or negative.

Is the question novel?

If a quick search in PubMed finds that other people have answered this question before, you should really consider whether it's worth pursuing. A good literature review before starting a project can save you a lot of time and effort. Some of my students have asked me "what do you mean by novel?" or "how novel do my findings have to be?" My typical answer is that a research question must be novel in at least one aspect - the population, the exposure, the outcome, the database, or the timing. If any of these components haven't been studied before, then it's new.

Let's take one quick example. Let's say there was a study published that found that in low-income adults, use of audio-only telehealth in 2021-2022 was associated with a lower no-show rate. There would be multiple ways to ask this question in a novel way. You could modify the population to be high-income adults or low-income children. You could modify the exposure to be video telehealth visits. You could modify the outcome from no-show rate to late-show rate. You could modify the database. Or you could modify the timing to be 2020-2025.

Once you've confirmed that the research question is clearly defined, interesting, and novel, your next step is to evaluate how the study will be conducted. The study design will determine both the strength of your conclusions and the amount of effort required from you.

The Study Design

Is the study prospective or retrospective?

Prospective studies collect new data that haven't yet been generated, while retrospective studies collect previously generated data. Each has their pros and cons. Prospective studies are best suited to answer a research question that cannot be answered with other existing data. Because no data has yet been collected, the study can be designed to guarantee that the study question can be answered. Furthermore, prospective studies are the only type of study that address causation, hence why all new medication studies are prospective studies. The major drawback of prospective studies is the effort and time needed to collect the data.

Retrospective studies are best suited to identify associations between exposures and outcomes. Because the outcomes have already occurred in the past, these are typically less time-consuming than prospective studies. The main drawbacks of retrospective studies are the inability to draw causal conclusions between exposure and outcome. All studies will also have residual confounding that cannot be accounted for in the study design, so the conclusions may also be less accurate. As a result, getting a high-impact journal to publish a well-designed retrospective study can be slightly more challenging than a well-designed prospective study.

If retrospective, what is the methodology of data collection?

If the main method of data collection is chart review, I would strongly consider trying a different type of project. Chart reviews are time-consuming, labor-intensive, and often have a very difficult time getting published in a high-impact journal. Chart reviews often base their results from a single center, which makes the findings much less generalizable to the rest of the journal's readership. The findings are also imprecise because of inter-rater reliability issues that commonly occur during data collection. However, there are a few situations where chart review can be appropriate. These include the identification of novel associations emerging from a new disease (i.e., COVID), associations based on objective physiologic measures that likely extend beyond the single center where the data was collected, or associations in extremely rare conditions where a large database has not yet been established.

The best type of study to work on is one where the data has already been collected and validated. This can mean studies based on publicly available datasets, previous clinical trials, or previously built databases. Instead of spending time on manual data collection, you can focus on actually doing the science.

Regardless of whether your project is prospective or retrospective, the quality and characteristics of your dataset will ultimately determine the impact and publishability of your findings. Here are the key dataset considerations that will influence your project's success.

The Dataset

What is the dataset's sample size?

Datasets need to be big enough to detect meaningful associations. It's difficult to say how big a dataset's sample size needs to be without a dedicated power calculation. In general, the sample size will need to be bigger when the associations are weaker, and vice versa.

Additionally, almost every study will likely need to be analyzed using regression. One rule of thumb is that you need 10 observations for every variable adjusted for in the linear regression model. So if you have 5 variables you need to adjust for, your sample size should be at least 50.

Can the findings be extrapolated to other contexts, or is it limited to the institution where the study was being conducted?

This is an intrinsic part of the dataset. Based on how the data was collected, the findings can be generalizable outside of the study site or not. An example of a non-generalizable study might be - "In this Chicago hospital, is the hospital's interdisciplinary team meeting associated with reduced readmission rate?" The hospital's team meeting might only work because of its location or the exact team. An example of a generalizable study might be "In this Chicago primary care clinic, is early initiation of influenza antivirals associated with lower hospitalization rates?" These findings might be generalizable to clinics outside of Chicago because it focuses on pathophysiology shared among humans rather than implementation of systems at a single hospital.

It's important to know the generalizability of the study before you start because it impacts the ultimate target journal of the paper. Studies that are more generalizable are more likely to end up in higher-tier journals, whereas non-generalizable studies might be more difficult to publish, even in middle-to-lower tier journals.

If this is a non-randomized study, does the planned dataset or existing dataset have all the variables you need to adjust for confounding?

This is also part of the dataset. You need to ensure that the dataset you choose is able to adjust for as much of the residual confounding as possible. Before committing to a project, request a data dictionary or list of available variables. Look specifically for demographic variables (age, sex, race/ethnicity, socioeconomic indicators), comorbidity measures (Charlson Comorbidity Index, individual diagnoses), and other factors that might influence both your exposure and outcome. If key confounders are missing, your study may be criticized during peer review for having unmeasured confounding, which can lead to rejection even if your analysis is otherwise sound. The strongest retrospective studies are those that can convincingly argue they've accounted for the most important confounding variables.

By carefully evaluating the research question, study design, and dataset before committing to a project, you'll dramatically increase your chances of achieving a successful publication. Remember, the time you spend evaluating potential projects upfront will save you months or even years of effort on projects that were doomed from the start.