The review process established by the What Works Clearinghouse (WWC) is comprehensive and systematic. There are three initial steps before reviewing studies within a topic:
- organizing a review team composed of a Principal Investigator, a Project Coordinator, and research analysts;
- creating a work plan; and
- developing a protocol that tailors the WWC Evidence Standards to the subject of each review.
Once a topic is identified, studies are gathered through an extensive search of published and unpublished research literature, including submissions from intervention developers and the public. The collected studies that meet broad relevancy and methodology criteria are first screened on the following dimensions: (a) the relevance of the intervention of interest, (b) the relevance of the sample to the population of interest and the recency of the study, and (c) the relevance and validity of the outcome measure. Studies that do not meet one or more of these criteria are categorized as "Does Not Meet Evidence Screens."
Second, the WWC determines the evidence of causal validity of each study according to WWC Evidence Standards and gives each study one of three possible ratings: "Meets Evidence Standards" (for randomized controlled trials and regression discontinuity studies that provide the strongest evidence of causal validity), "Meets Evidence Standards with Reservations," (for quasi-experimental studies; randomized controlled trials that have problems with randomization, attrition, or disruption; and regression discontinuity designs that have problems with attrition or disruption), and "Does Not Meet Evidence Screens" (for studies that do not provide strong evidence of causal validity).
Third, studies that "Meet Evidence Standards" and "Meet Evidence Standards with Reservations" are reviewed further to describe and rate other important characteristics. These characteristics include: (a) intervention fidelity, (b) outcome measures, (c) the extent to which relevant people, settings, and measure timings are included in the study, (d) the extent to which the study allowed for testing of the intervention's effect within subgroups, (e) statistical analysis, and (f) statistical reporting.
The WWC regularly updates WWC technical standards and their application to take account of new considerations brought forth by experts and users. Such changes may result in re-appraisals of studies and/or interventions previously reviewed and rated. Current WWC standards offer guidance for those planning or carrying out studies, not only in the design considerations but the analysis and reporting stages as well. WWC standards, however, may not pertain to every situation, context, or purpose of a study and will evolve.
The WWC has a two-tiered reporting system that generates reports on the intervention and topic level.
Intervention Reports: Intervention reports are produced for interventions that had one or more studies that met WWC Evidence Standards. The reports provide key findings from each of the studies pertaining to the particular intervention. Each report describes the intervention (for example, program, product, practice, or policy) and has a brief description of each outcome study. The report also presents in a single table the findings from the WWC-vetted studies. These reports are released as soon as they are produced, typically at the same time as the topic report. Intervention reports cannot be prepared for interventions whose studies do not pass WWC Standards.
Topic Reports: Each topic report briefly describes the topic and each intervention that the WWC reviewed. The report covers only interventions that had studies passing WWC Standards. Topic reports are usually released at the same time as intervention reports because each topic level report provides a compilation of completed intervention reports. The topic report describes how the WWC searched the literature, describes the key features of interventions at the time they were studied, and presents the findings. The topic report also notes the over-all strength of the research base for each intervention, providing an accessible picture of interventions that met WWC Evidence Standards. The topic report links to all related intervention reports.
WWC Review Team
The What Works Clearinghouse has assembled high-quality review teams for each WWC topic. Each team consists of a Principal Investigator (PI), a Project Coordinator (PC), and research analysts.
The PI for each topic is a well-known expert in his/her field and is responsible for leadership in conceptualizing the specific topic area, identifying and addressing issues during the review, and developing and reviewing topic and intervention reports developed for the topic. Leadership includes overseeing the quality in the production of the reports and making decisions, based on methodological and substantive expertise, that are not otherwise covered in the WWC protocols and procedures for report production.
The Project Coordinator (PC) is an established researcher with relevant methodological and substantive expertise. The coordinator oversees the work of the WWC Review Team, manages that specific review, reviews research ratings, and writes and revises the work plan, protocol, and draft and final reports in collaboration with the PI.
Research analysts have experience conducting critical reviews of research and have training in research design and methodology that is relevant to reviewing evidence of effectiveness. As part of the review team, research analysts review and summarize the evidence of effectiveness.
Quality control is built into every stage of the review process. All reports undergo extensive review, including the following (actual quality control measures differ for topic and intervention versus study coding results): an opportunity for researchers who conducted the original study to review the study coding results, an opportunity for the intervention developer to review the intervention report, and review by the WWC Steering Committee (task leaders in the contracting organizations).