Evaluating Evidence and Research

When it comes to exercise physiology, coaches, authors, apps and training plans constantly talk about how they are 'science-based' or 'research-backed.' And hopefully, they are!

Yet while scientific research is helpful in starting to determine training methodology, it's not the whole picture, and pretending it is can be damaging.

There are a few core principles that are important to understand when evaluating evidence, from anecdotes and elite training schedules, to large peer-reviewed studies.

What Strengthens Evidence

No matter the type of evidence, there are a variety of factors you should look at to assess how applicable the findings are to your training.

• Type of evidence (anecdote, observational research, experimental trial)

• How many subjects?

• Who were they? (How similar are they to you)

• How long a period?

• Has it been replicated?

The holy grail of research in terms of applicability to you would be several experimental trials reaching similar conclusions, on a large number of subjects that are similar to you, for a long period of time.

In reality, all of us need to synthesize evidence of various types, because there is almost none that meets all of those criteria!

The Limits of Anecdote

Let's get one thing out of the way. Social media can give you lots of people saying 'doing x worked for me,' and sometimes, even a chorus of people saying that.

It's still not evidence. Here's the thing—social media algorithms only show you a tiny, filtered slice of a tiny slice. Engagement-ranked feeds show you posts you're most likely to engage with that are already getting engagement. The result is that the post reaches those most likely to agree with it, and comment saying so. Those who didn't try x, or tried it and got injured, or just didn't make progress—they either don't see the post at all, or don't bother to comment. The failures are totally invisible.

Another commonly used piece of evidence is the training schedule of an athlete significantly above your level, whether that's a world-class runner or just someone a lot faster than you. That's anecdotal evidence of the least practical kind, because you're seeing the result of their fitness, not the cause, and all that's wrapped up in golden genetics and a decade of training experience, not to mention expert coaching and the ability to sleep nine hours a night. Copying the training schedule of someone else almost never pays off.

Correlation ≠ Causation

This is a crucial aspect, especially when evaluating anecdotal evidence and advice from friends (or the internet). You will find a success story for just about any training methodology or lack thereof. You will find runners advocating for one long run per week, and an all-out mile mid-week, with no other physical activity, and swearing it works for them.

And it might have. But did that runner succeed because of what they did, or in spite of what they did? We don't know. That's what scientific research is for, but of course then it's subject to the caveats below.

The Limits of Research

Scientific studies can be a powerful tool for understanding exercise physiology, but reading a study well means knowing what it can and can't tell you.

A study answers a narrow question about a specific group of people over a specific window of time. The narrowness is what makes it possible to get a clean answer.

But often, the conclusions of research get carried out of the original narrow frame and applied as a general rule to populations that they may not hold true for.

And—important to note—most of what runners experience of these research studies is articles, podcast summaries or a coach citing them. Each often drops qualifiers, which would otherwise tell you whether the finding even applies to you.

Types of Research

There are two types of research on runners: observational and experimental.

Observational studies do just that—observe. That could be a large number of people at a single point in time, such as a single race, or a smaller segment over a much longer period of time such as several years.

While this is ideal for looking at the real-world benefits of the studied variable, it also introduces a problem: the lack of causation. Observational studies can only show that two variables go together (correlation), but cannot show causation. For example, a study can show that faster runners tend to run higher mileage. But is it the mileage that makes them fast, or do fast runners simply tolerate more miles and cover more distance in the same time?

Or, are faster runners more likely to be later in their careers, having already worked through, and fully adapted to lower volume levels? That's a confounding variable, or an uncontrolled, unstated variable that affects both the others.

By contrast, experimental trials isolate a specific variable and attempt to determine causation by randomly assigning subjects to groups so confounding variables spread evenly across them. However experimental studies are generally short duration and on relatively few subjects. You'll often see experimental studies of a few dozen subjects or fewer, and a time frame of just a few weeks to a few months.

The reason experimental trials are short and small comes down to practicality. Controlling a bunch of athletes' training and measuring their progress for months on end is expensive and difficult. It's often hard to even recruit for these trials, because they interrupt an athlete's training and racing.

So, trials will frequently recruit untrained subjects, historically university students. Primarily young, male, recreationally active subjects—those most primed to reap the benefits from any sort of novel stimulus.

The trial also needs to be relatively short, or subjects get busy, sick, hurt or otherwise just drop out of the study.

Short trials primarily catch early adaptations, which are often the fastest and strongest. They also can miss negative aspects of the studied variable such as injuries or regressions, or even just the cost of losing whatever the new variable replaced.

The two kinds of research both have blind spots: one can look across thousands of people and see long timelines, but can't tell cause and effect. The other can show causation, but only over a short window of time in relatively few people.

There's one more thing worth asking of any new finding: has any other study found the same thing? A single study is a hypothesis with some evidence attached, not a settled answer. Findings that were solid in one trial often shrink or vanish when someone runs it again with different people. Several studies pointing the same direction, on different groups, is worth far more than one dramatic finding.

Who Was Studied

As mentioned above, a study is narrow, and its conclusions apply to the population it was run on. Whether it expands to cover other groups, and how far that expansion goes, is an open question.

Remember those young, male, recreationally active subjects? What if you're a 48-year-old female who's eight years into her running—do the findings from a group of college kids still apply to you?

The biggest gap is the subjects' training age. An untrained subject, or someone with fewer miles in their legs, adapts far differently than a lifelong athlete. An intervention shown to work in one group won't necessarily hold true for the other.

There's still a problem for studies of people with longer training ages: the people in the study are the ones who actually made it into the study. Research on age 50+ runners measures those still running past age 50. The ones who got hurt, burned out, or quit aren't in the data, and neither are their reasons. The same goes for any study of experienced athletes. You're looking at the group whose bodies tolerated whatever they were doing.

There are lots of other aspects to consider as well. Consider two athletes with the same training age, one who worked primarily on 5K to half marathon events, and the other who trained for marathon and ultramarathon events. The same intervention likely won't produce the same results for both.

None of this means we should ignore or dismiss the science entirely, it just means we need to contextualize the findings.

A Study of One

Experimental trials, by design, isolate and study a single variable. But exercise physiology is incredibly complex: a variety of physiological factors with varying amounts of trainability, interacting with genetics, mechanical systems and movement patterns.

Our training by necessity is even more complex, as it lives at the intersection of physiology, psychology and real life context.

While a variable might be shown to have a certain effect in a study, we often don't know how that variable interacts with all the other variables, or when presented with all of the individual variations mentioned in the section above.

While research studies are very useful, in reality, you are a study of n=1. Your variables—history, genetics, life context and more—are unique. How the science is applied to you likely differs from how it's applied to someone else.

The further you are from the study subjects, the more you should treat the finding as something to experiment with and see if it works, rather than a rule to follow.

Where They Meet

Research and anecdotal information answer different questions. Research tells you what happened to a defined group under defined conditions. That's a great starting point, and it tells you which ideas are even worth your attention. What it can't tell you is what will happen to you.

Anecdotal evidence, done right, can start to fill in the gaps. It expands the research with information on more and varied people, with more variables. That requires more than a single comment thread though; it's the volume of experience that matters.

This is where anecdotal information from a coach, or from many coaches, can shine. One athlete's experience is a single data point. Watching the same principle play out across dozens or hundreds of athletes over years is a pattern, and coaches who know their history and context can fill in a lot of gaps. It covers the ground research has the hardest time reaching: long timelines, trained bodies, and the complexity of a human life.

The final step though, is to learn how you respond to a given training intervention, with all the context of your history, genetics and life. Research and anecdotal information can tell you what's likely to work; only your own response tells you what did.

Approach both with an educated eye, gather as much data as you can, and keep revising what you think you know. Your training should adapt based on what actually happens, not stay locked to what you expected.