The Diet Doctor policy for grading scientific evidence

 
As we base our guides on scientific evidence, it is important to have a clear policy for how to grade the strength of different kinds of evidence. Our policy is in many ways similar to other documents of its kind.1 We define the levels of evidence as follows:

  1. Systematic reviews of multiple, high-quality randomized controlled trials:2 Strong evidence.
  2. Randomized controlled trials (RCTs): Moderate evidence. If repeated, very clear and consistent results over multiple trials: Strong evidence.
    Non-randomized controlled trials: Weak evidence. Note that while non-randomized trials are weak for determining the best intervention, they can provide other unique insights.3
  3. Observational/epidemiological data is only seriously considered if the hazard ratio (HR) is over 2.4 With an HR below 2, evidence shows that the correlation is likely random and false,5 and thus it’s more likely to confuse than to inform: it is not causal evidence.
    With an HR > 2 in high-quality prospective cohort studies: Weak evidence.
    With an HR > 2 in lower-quality observational studies: Very weak evidence.
    Under exceptional and rare circumstances with HR consistently > 5 in several high-quality observational studies, with biological plausibility, no other obvious explanation and generally following the classic Bradford Hill criteria:6 Moderate evidence (e.g. smoking and lung cancer).
  4. Consistent clinical experience from several experienced practitioners is also considered, as long as there’s no high-quality science (i.e. randomized intervention studies) to contradict it: Weak evidence.
  5. Case reports and anecdotes are evaluated with caution, and not used as evidence unless there’s a lack of higher-quality evidence available: Very weak evidence.
  6. Animal studies are not considered in fields where there already exist studies on humans. If it’s the only evidence available: Very weak evidence.
  7. Opinions of world-leading experts: No evidence. Opinions are not evidence, no matter who has the opinion. To be evidence-based, everyone has to support his or her opinions and theories with believable facts, i.e. scientific evidence.

 


Success stories

As mentioned above under “case reports and anecdotes”, success stories are considered very weak evidence when it comes to determining if a particular lifestyle change is beneficial. This is for many reasons, perhaps primarily that these stories come from a selected population of successful people. We don’t know how many people may have attempted the same lifestyle change without being happy with the results, or only had more moderate success.

However, these stories can add value when it comes to more deeply understanding the subjective experience and feelings of a selected group. This can potentially add another level of insight and inspiration that numbers and statistics alone can’t provide.7


Financial bias

Evidence from studies at high risk of financial bias, e.g. studies on a drug funded and conducted by the company selling the drug, is discounted by one level. Studies suggest that this financial bias may be strongly influencing the reported outcome.8

Nutritional studies funded by companies with a financial interest in the outcome are likewise discounted.9


Evolutionary considerations

Consistency with what is evolutionarily probable can strengthen evidence, and inconsistency can weaken evidence.

For example, humans and their ancestors have been eating natural saturated fats for millions of years. It’s evolutionarily unlikely that eating it in amounts roughly similar to before is the cause of a new epidemic of chronic disease.

On the other hand, refined pure sugar in large quantities is a phenomenon of the last 150 years. It’s evolutionarily possible that it could have negative health consequences, as humans may not yet be fully adapted to it.

While these examples do not prove cause and effect, they do add context to the scientific evidence and can therefore help elevate or decrease the level of evidence somewhat.

This line of reasoning might also be applied to long-term treatments with drugs that reduce the function of an enzyme or receptor in the body. If that was a good idea, all things considered, it could have been relatively easy for evolution to remove or reduce that function (or never develop it in the first place). This means that the positive effect of the drug might be outweighed by side effects under most circumstances for most people. Thus long-term treatment with a drug might be a good idea — if it is a good idea — primarily under specific circumstances and/or for people with specific diseases.10

From this perspective it can also be argued that interventions based on eating certain natural foods — or avoiding foods for a short time (e.g. for one day) — are things that the human body should be evolutionarily adapted to, to a large degree. These interventions may thus be significantly safer, compared to new drugs or surgical interventions.

The evolutionary lens as a way to strengthen or weaken evidence has to be used with a large dose of caution, as it’s not always apparent what environmental factors our ancestors were exposed to, and many of these factors have varied widely over time and geography.


Core references

 
Advances in Nutrition 2018: Limiting dependence on nonrandomized studies and improving randomized trials in human nutrition research: why and how

JAMA 2018: The challenge of reforming nutritional epidemiologic research

PLoS Medicine 2005: Why most published research findings are false


More

The Diet Doctor policy for evidence-based guides

The science of low carb and keto

Comments

Do you have comments on, objections to or suggested changes or additions to our guidelines? Feel free to leave a comment below.

  1. For example, our Diet Doctor policy is similar to the Oxford Centre for Evidence-based Medicine – Levels of Evidence document.

  2. Wikipedia: Systematic review

    Wikipedia: Meta-analysis

    Wikipedia: Randomized controlled trial

  3. The weakness is primarily related to the unfair start – it’s very hard to know which of the tested interventions was the best without a randomized trial.

    However, if an intervention has an effect size that is far larger than what is normally expected, that is still a good indication of what results can be expected for a self-selected, real world population.

    A non-randomized intervention study may add complimentary data on expected effect size that can’t be found in RCTs of non-self-selected participants, especially when the RCT is using intention-to-treat analysis. The RCT will likely give significantly lower numbers for the average effect size, compared to what can be expected for a motivated patient who manages to follow a prescribed lifestyle intervention.

    A non-randomized trial may give the best estimation of the average effect size for someone who is managing to follow a specified lifestyle treatment.

  4. An HR > 2 means that a (lifestyle) factor is associated with at least twice (double) the risk of something.

    Wikipedia: Observational study

    Wikipedia: Hazard ratio

  5. Advances in Nutrition 2018: Limiting dependence on nonrandomized studies and improving randomized trials in human nutrition research: why and how

    JAMA 2018: The challenge of reforming nutritional epidemiologic research

    PLoS Medicine 2005: Why most published research findings are false

  6. Wikipedia: Bradford Hill criteria

    Proceedings of the Royal Society of Medicine 1965: The environment and disease: association or causation? By Sir Austin Bradford Hill

  7. Furthermore, there is occasionally a “black swan” element to these stories. For example, if a condition is generally considered chronic and progressive, like type 2 diabetes, and hundreds of people share stories about reversal of the disease using nothing more than a lifestyle intervention, this raises a legitimate question: Is the common view true, occasionally not true, or simply false?

    This question will then have to be answered using higher-quality scientific evidence such as controlled intervention studies.

  8. BMJ 2017: Financial ties of principal investigators and randomized controlled trial outcomes: cross sectional study

    PLOS ONE 2016: Relationship between research outcomes and risk of bias, study sponsorship, and author financial conflicts of interest in reviews of the effects of artificially sweetened beverages on weight outcomes: A systematic review of reviews

    BMJ 2003: Pharmaceutical industry sponsorship and research outcome and quality: systematic review

  9. There are also other sources of bias in nutritional studies, that may or may not be disclosed, and that can have an influence. An example of such bias may be strongly held dietary preferences:

    JAMA 2018: Disclosures in nutrition research. Why it is different

    To some degree this potential bias may have to be taken into account when interpreting the strength of evidence of a study, although it may be harder to quantify than financial bias.

    We require our authors and reviewers of evidence-based guides to disclose any clear potential source of bias, not just financial bias.

  10. The counterargument to this is that our environment may have changed so much that what was evolutionarily beneficial thousands of years ago (or more) might occasionally be neutral or even harmful today. For example, a genetic adaptation that protected against infections but had other side effects might be negative today, in an era of antibiotics.

    Sometimes our environment may have changed so much that long-term treatment with a drug that reduces a natural function could potentially be a good idea.

    However, the safest course of action might often be to first try to more closely emulate the environment that was common before, at least when it comes to safe things like eating natural old-fashioned foods, exercising, getting adequate sunlight, sleeping enough, etc. For example, this might reduce the need for long-term treatment with drugs for components of the metabolic syndrome, like type 2 diabetes.