How to Select an Evidence-Based Prevention Intervention for My Community

In our other Nuggets we focused on defining prevention science and its application to practice.  We introduced the work of the United Nations Office on Drugs and Crime that reviewed the research literature and summarize evidence-based strategies and principles by age group but also by delivery setting (family, school, workplace, community including entertainment venues, and health care settings) in the International Standards .  These strategies and principles are summaries of rigorous evaluations conducted of specific manualized programs.  Although the Appendices provide the sources of the research, registries are more helpful to those seeking assistance in selecting an evidence-based intervention to meet the specific needs of our communities.

As evidence-based prevention interventions have a theoretical foundation, they are generally composed of several key constructs or components such as beliefs regarding the normative nature of substance use, perceptions of harms or risks associated with substance use, or having specific skills such as communication skills.  In addition, evidence-based prevention interventions are structured to enhance learning that is relevant to the age-group being addressed.  Although current prevention researchers are working to determine what are the components of an intervention that are statistically the most important ones associated with positive outcomes, much more work must be accomplished before we can talk about key components of an intervention.  So until we have that information, most interventions/programs should be delivered as they are designed. Where can someone interested in implementing an evidence-based intervention find programs that meet the needs of their community. They can review the research literature, although that is time consuming.  Or they can seek the information they need from a register.

So let’s talk about prevention registries.  What are they? Where are they? How do they differ? How do I use them?

Registries are not just lists of prevention programs. They include manualized and, in some cases, environmental prevention programming that:

• Meet specific criteria of research evidence

• Involve a review process using groups of independent raters.

  
  

Registries vary considerably in these criteria, in guidelines for the independent reviews, as well in the qualifications of the raters. Registries are available at the International level (e.g., Xchange Prevention Registry of the European Union Drugs Agency), National level (e.g., BluePrints, Crime Solutions), and at the state level (e.g., The California Evidence-Based Clearinghouse). The urls for these sites are provided below.

Most registries include programs that meet the most rigorous criteria and those that do not meet all of these criteria but are still considered very effective. For example:

Blueprints rates interventions as either Promising, Model or Model Plus. The criteria for these categorizations are:

Promising interventions meet the following standards:

- Intervention specificity: The intervention description clearly identifies the intended outcome(s), whether specific risk and/or protective factors are targeted to produce this change, the population for which the intervention is intended, and how the components of the intervention work to produce this change.

- Evaluation quality: The evaluation trials produce valid and reliable findings. This requires a minimum of (a) one high-quality randomized control trial or (b) two high- quality quasi-experimental evaluations.

- Intervention impact: The preponderance of evidence from the high-quality evaluations indicates a significant positive change in intended outcomes that can be attributed to the intervention and there is no evidence of harmful effects.

- Dissemination readiness: The intervention is currently available for dissemination and has the necessary organizational capability, manuals, training, technical assistance and other support.

Model interventions meet the Promising intervention criteria PLUS:

- Replication: A minimum of (a) two high-quality randomized control trials or (b) one high-quality randomized control trial plus one high-quality quasi-experimental evaluation.

- Long-term follow-up: Positive intervention impact is sustained for a minimum of 12 months after the program intervention ends.

Model Plus interventions meet Promising and Model standards PLUS:

- Independent Replication: In at least one high-quality study demonstrating desired outcomes, authorship, data collection, and analysis has been conducted by a researcher who is neither a current or past member of the program developer’s research team and who has no financial interest in the program.

---

The California Evidence-Based Clearinghouse uses a different rating scale for programs and practices that are listed in the registry. This scale is from 1 to 5. A scientific rating of 1 represents a practice with the strongest research evidence and a 5 represents a concerning practice that appears to pose substantial risk to children and families. Some programs do not currently have strong enough research evidence to be rated on the Scientific Rating Scale and are classified as NR - (Not able to be Rated). 

• Well-Supported by Research Evidence

- At least 2 rigorous randomized controlled trials (RCTs) with non-overlapping analytic samples that were carried out in the usual care or practice settings have found the program to be superior to an appropriate comparison program on outcomes specified in the criteria for that particular topic area

- In at least one of these RCTs, the program has shown to have a sustained effect of at least one year beyond the end of treatment, when compared to a control group.

• Supported by Research Evidence

- At least one rigorous RCT in a usual care or practice setting has found the program to be superior to an appropriate comparison program outcomes specified in the criteria for that particular topic area.

- In that RCT, the program has shown to have a sustained effect of at least six months beyond the end of treatment, when compared to a control group.

• Promising Research Evidence

- At least one study utilizing some form of control (e.g., untreated group, placebo group, matched wait list) has done one of the following:

- established the program's benefit over the control on the outcomes specified in the criteria for that particular topic area,

- found it to be comparable on outcomes specified in the criteria for the topic area to a program rated 3 or higher on this rating scale in the same topic area,

OR

- found it to be superior on outcomes specified for that particular topic area to an appropriate comparison program. Promising Research Evidence

• Evidence Fails to Demonstrate Effect

- Two or more RCTs with non-overlapping analytic samples that were carried out in usual care or practice settings have found that the program has not resulted in improved outcomes specified in the criteria for that particular topic area, when compared to usual care.

- The overall weight of evidence does not support the benefit of the program on the outcome specified in the criteria for that particular topic area.

• Concerning Practice

- One or more of the following statements is true:

- if multiple outcome studies have been conducted, the overall weight of evidence suggests the program has a negative effect on the target population being served or on outcomes specified in the criteria for that particular topic area;

- there is case data suggesting a risk of harm that: a) was probably caused by the program and b) was severe and/or frequent;

OR

- there is a legal or empirical basis suggesting that, compared to its likely benefits, the program constitutes a risk of harm to those receiving it.

• NR - Not Able to be Rated

- The program does not have any published, peer-reviewed study utilizing some form of control (e.g., untreated group, placebo group, matched wait list study) that has established the program's benefit over the control on outcomes specified in the criteria for that particular topic area or found it to be comparable to or better than an appropriate comparison program on outcomes specified in the criteria for the topic area.

- The research evidence for programs listed NR cannot be established at this time. This does not mean the programs are ineffective, but rather that there is not research evidence that meets the criteria for any other level on this rating scale.

---

Xchange Prevention Registry uses different terms for its ratings. Only interventions that target substance-related/violence/delinquency outcomes are included in the registry. Then:

• Is the intervention still active or able to be used in Europe?

• Is the intervention clearly defined (outcomes, target group, risk and protective factors, logic model or theoretical rationale)?

• Is there at least one suitable evaluation study in Europe (RCT, Quasi-experimental design or interrupted time series) for this intervention?

• Are the expected outcomes measured?

If the interventions meet these criteria they are then rated by independent raters using the following categories:

• Beneficial: Interventions for which convincing, consistent and sustained effects for relevant outcomes are in favour of the intervention as found in two or more studies of excellent quality in Europe.

• Likely to be beneficial: Interventions for which convincing and consistent effects for relevant outcomes are in favour of the intervention as found in at least one evaluation study of excellent quality in Europe.

• Possibly beneficial: Interventions for which some effects for relevant outcomes are in favour of the intervention as found in at least one evaluation study of acceptable quality in Europe. An intervention ranked as ‘possibly beneficial’ is suitable for application in the context of more rigorous evaluations.

• Additional studies recommended: Interventions for which concerns about evaluation quality or consistency of outcomes in Europe make it difficult to assess if they are effective or not, even if outcomes seem to be in favour of the intervention.

• Unlikely to be beneficial: Interventions for which at least one evaluation of excellent quality in Europe shows convincing evidence of no or harmful effects on relevant outcomes.

• Possibly harmful: Interventions for which some effects for relevant outcomes of the intervention are considered harmful, as found in at least one evaluation study of acceptable quality in Europe. An intervention ranked as 'possibly harmful' is unsuitable for application except within a framework of other priorities and with rigorous and strictly supervised evaluations.

By and large there is overlap across the registries for many of the stronger prevention programs.  The challenge is to find programs that fit the needs of your target population or community.  The registries provide the tools to select programs by outcome, by the characteristics of the target population, by setting, or by risk and/or protective factor.

Once you enter your information, the registries provide a description of the program and of the evaluation research and contact information to learn more about possible training and accessing materials to deliver the intervention.

Accessing Registries:

Blueprints for Healthy Youth Development. Blueprintsprograms.org

The California Evidence-Based Clearinghouse. https://www.cebc4cw.org/

Xchange Prevention Registry. https://www.emcdda.europa.eu/best-practice/xchange