Systematic reviews are often utilized as a rigorous way of summarizing the best available evidence on a topic. However, when it comes to health policy, systematic reviews are not always a feasible or an appropriate option. Instead, alternative methodologies are often chosen, including for example, rapid reviews, expert opinion, and grey literature reviews. The rigor of the methodology used to compile evidence can impact research findings, conclusions and recommendations; but while there well-established standards for conducting systematic reviews (i.e. the Cochrane handbook), the less rigorous methodologies are currently lacking such established standards.
At the Cochrane Canada Annual Symposium (2015), Brittany Gerber of Medlior Health Outcomes Research Ltd., presented her insights into the methodology of evidence-based reports published by health care organizations in Canada, the US and UK.
Nineteen reports were selected from the following organizations:
- Canadian Agency for Drugs and Technologies in Health;
- Health Canada;
- Public Health Agency of Canada;
- National Institute for Health and Care Excellence; and
- Agency for Healthcare Research and Quality.
Medlior analyzed the selected reports to assess whether they included the methodological details required for a Cochrane systematic review, for example:
- Types of studies, participants, interventions, comparators and outcome measures (PICO)
- Search methods
- Electronic Searches & Other Resources
- Study Selection
- Assessment of Bias
- Data extraction
The transparency in methodological details varied across the 19 reports. Only 13 reports described their methods and of these 6 reports provided adequate detail. The majority of the 13 reports did not include other important details such as: search strategies, selection of criteria or quality assessment. The differences in methodological details between the selected reports not only varied across the organizations listed above, but often within organizations as well.
The findings suggest that there is a need for clear and consistent standards for reporting the methodology of evidence-based reports in health care. Drawing on the transparency emphasized by the Cochrane handbook, we believe that all evidence-based reports should clearly state pertinent methodological details so that audiences are able to understand how the research was conducted. This would allow the readers to better understand any biases or limitations of the report, to further inform their interpretation of the evidence.
To increase the validity of evidence-based health care reports, a transparent methodology should be reported to ensure meaningful and appropriate decision-making.