Pragmatic trials: real-world settings, real-world solutions
Randomized controlled trials (RCTs) are considered the gold standard for studying the safety and efficacy of new or existing health interventions like drugs, devices or technology. However, traditional RCTs test these interventions under optimal conditions, so that the findings of these RCTs may not be true when these interventions are applied in usual care, in non-research settings or to patients’ everyday lives. Attempts to address this have led to pragmatic trials, which are designed to evaluate the effectiveness of interventions in routine practice and are critical to inform decision-making by patients and their families, physicians, administrators and policy-makers in real-world settings.
A team led by ICES Western scientist Dr. Merrick Zwarenstein developed a tool for designing randomized trials that would be more pragmatic. The tool, PRECIS-2, guides trial designers in building randomized trials whose findings are more relevant to decision makers. The tool uses nine domains — eligibility criteria, recruitment, setting, organization, flexibility (delivery), flexibility (adherence), follow-up, primary outcome and primary analysis — each ranked on a scale from 1 (explanatory, or “ideal conditions”) to 5 (pragmatic, or “real world”). Using the tool helps ensure trials are designed so that their results are relevant to, and used by, patients, clinicians and policy makers. In the early 2000s, no pragmatic trials were being done in Ontario. Twenty years later, dozens of pragmatic trials using PRECIS-2 and data available at ICES have been conducted, combining the benefits of pragmatic attitudes with randomized trials and vast administrative data sets.
Researchers have found a substantial benefit to conducting pragmatic trials using ICES databases. Instead of identifying and testing an intervention on a small sample of people with a particular condition, researchers are able to look at the data on everyone with the condition in Ontario and conduct the trial on all of them very cheaply because data
collection costs are low. These pragmatic trials are both less expensive than traditional RCTs and more comprehensive.
How this work is having impact
- Papers describing the suite of tools for designing randomized trials to be more pragmatic have been cited over a thousand times and are widely regarded as the standard descriptions of the pragmatic approach to randomized trial design.
“Unfortunately, most traditionally designed randomized controlled trials have small samples because recruitment and data collection are expensive. Also, their findings are difficult to apply to real-world questions as patients, clinicians and settings are highly selected. Because ICES data are easily and cheaply accessible and represent all Ontarians, they can be used to conduct precise, widely applicable and low-cost pragmatic RCTs that provide policy-relevant answers on the effectiveness, safety and efficiency of many clinical, public health and social policy choices.”
Dr. Merrick Zwarenstein, Senior Scientist at ICES Western
- Between them, Dr. Zwarenstein and Dr. Baiju Shah have led five
large-scale pragmatic randomized trials using ICES data to evaluate
the effect of knowledge translation interventions on patients of
Ontario primary care physicians and home care providers. These studies found that at a large scale, different kinds of
educational materials did not improve family physician adherence to evidence-based
care guidelines, and that care pathways
for home care provision
did not improve care outcomes
for a number of conditions.
- The ISLAND trial, led by Drs. Noah Ivers and J.D Schwalm, showed that interventions like mail-outs and phone calls encouraged more patients to adhere to their cardiac rehabilitation program after a heart attack.
- The MyTEMP trial, led by Drs. Ahmed Al-Jaishi and Amit Garg,
is examining the effect of patients being provided a personalized
dialysate temperature during each dialysis session on their risk of
cardiovascular-related death or hospitalization.
- The CHOICES trial, led by Dr. Jay Udell, uses a multicomponent
intervention and implementation approach that includes audit and
feedback reports for family physicians and educational materials
and tools for patients to help increase the use of cholesterol-lowering
Posted November 2020
Published in the 2019-20 Annual Report
Download the PDF
Read more impact stories