Appraising Evidence

The best clinical practice requires one to appraise and use the research evidence available while making clinical decisions. The skill to appraising scientific evidence will become easy if one takes a methodical approach.

In this article, I will provide you with steps to follow while appraising a scientific article. It is key to know the structure of a research article before you attempt to appraise the evidence.

Most research articles will include the following sections:

  1. Title
  2. Abstract
  3. Introduction / Background (leading to the need for research)
  4. Aims
  5. Methods / Methodology
  6. Results / Findings
  7. Conclusion
  8. Discussion
  9. References

Consider asking the following questions while appraising a research article. This is not an exhaustive list of questions to ask while appraising evidence but can be used direct your assessment.

I. Title:

  • Does the title cover the specific theme or the main aim of the research?
  • Is the title relevant to the aims of the research?
  • Does the title attract the readers while capturing the important theme of the study?
  • Does the title include the ‘PICO’ in a randomised controlled study?
    • Population / Problem / Patients (with disease in question)
    • Intervention / Exposure / Prognostic / Risk factor (under investigation)
    • Comparison / Control group intervention
    • Outcomes

II. Abstract:

  • Are the main features of all sections of the article mentioned briefly and clearly in the abstract?
    • Background: Is the ‘need for research’ mentioned?
    • Aims: Is the ‘main aim’ of the study mentioned?
    • Methods: Study design, randomisation/allocation methods, statistical analyses?
    • Results: Participant characteristics, results of the main hypothesis?
    • Conclusions: Description/explanation of the main results?
    • Discussion: Implication of the findings? How do these findings relate to the existing scientific literature?
  • Does the information packed in the abstract correspond to what is in the manuscript?

III. Introduction:

  • Does the introduction/background section clearly state what already is known about this topic?
  • Is the need for research clearly outlined?

IV. Aims:

  • Is the main aim of this study clearly stated?
  • Does the need for research from the background section and the main hypothesis (aim) of this study relate conceptually?

V. Methods:

  • Subject Selection Process: If the subjects are selected wrongly, the findings from the study risks having limited relevance and applicability to the population other than one sample.
    • Was the subject selection process described clearly?
    • Are the inclusion and exclusion criteria listed?
    • Was the population from which the subjects were selected defined clearly?
    • What is the subjects’ likelihood of truly representing the population that they were selected from?
    • What proportion of selected subjects completed the study?
    • Are the details of excluded subjects, missed cases, withdrawals, no-shows, lost to follow-up presented?
    • Can we compare the study sample to similar studies?
    • Can we generalise the study sample to other population?
  • Flow Diagram:
    • Is there a study flow diagram listing subjects included, excluded, missed, withdrawn, lost to follow-up and who successfully completed the study (in both experimental & control groups).
  • Did the study have sources of sampling bias while selecting subjects by following methods?
    • Convenience sampling (may favour/disfavour certain subjects)
    • Volunteer / Self-selection sampling (subjects may have similar characteristics)
    • Stratified sampling (may allow selection bias with unclear strata)
    • Random sampling (may result in unbalanced groups)
    • Other sampling methods: systematic, clustered, quota, snowball etc.,
  • Assessing Validity:
    • Did the study use measures/methods with adequate validity?
      • Face validity: Did the study use a measure/method that appears appropriate for the construct it is intending to assess?
      • Construct validity:
        • Did the study use a measure/method that estimates the theoretical construct adequately?
        • Does the measure/method used in the study correlate with other measures/methods that estimate the same theoretical construct adequately?
      • Content validity: Did the study use a measure/method that fully represents the construct it is intending to assess?
  • Assessing Reliability:
    • Did the study use measures (eg., bone mineral density by DEXA) with adequate reliability?
    • Did the study use methods (eg., using two radiographers for diagnosis) to increase the reliability of subjective assessments?
    • Did the study use methods (eg., taking serial MRIs) to increase the reliability of objective measurements (i.e., to reduce systematic/measurement errors)?

VI. Results:

  • Does this study require a statistical analysis? (Literature review vs Method study vs Empirical experiment)
  • Adequate statistics were done?
    • Did the authors report sample size calculation?
    • Did they report criteria for allocation into groups?
    • Did they use a clear unit of measurement?
    • Do the data/results address the research question?
  • Presenting data:
    • Are tables and figures understandable without reading the text?
    • Is there a repetition of content between tables/figures and text?
    • Is there adherence to common conventions on units, symbols, format, rounding etc.,?
  • Ensuring validity of data:
    • Did authors define statistical significance clearly?
    • Were the effect size, practically important differences explained?
    • Are the data/measurement categories used for analyses clearly defined and are they valid/reliable?

VII. Conclusion:

  • Does the conclusion section describe/explain/interpret the results succinctly?
  • Does the conclusion answer the main research question (aim) of this study?

VIII. Discussion:

  • Are the main findings re-stated briefly?
  • Are the results interpreted and compared with current literature?
  • Do the findings contribute to the existing body of knowledge?
  • Are the implications of the research mentioned?
  • Are the limitations of the study stated?
  • Find out if the limitations of the study contain serious errors? eg., wrong study design.
  • Are the future research directions suggested?

IX. References:

  • Did the authors use relevant articles for reference?
  • Did the authors use recent articles for reference?
  • Did the authors cite the referred articles correctly?
  • Can you find the cited articles in the online databases?
  • Did the authors fail to cite seminal articles for their area of research?