Research Approach

Download this document: Microsoft Word | Adobe PDF

The Access Center Research Continuum

With impetus from the No Child Left Behind (NCLB) Act (2001), which requires that states adopt and implement “scientifically based” programs and strategies, the Access Center was established to provide research-based strategies that help students with disabilities access the general education curriculum. Using the term “research based” to describe programs and strategies is not without ambiguity or controversy, however. As nationally recognized researcher Russell Gersten highlighted, “when the term ‘research says’ or ‘research supports’ comes up in a conversation about education, it is greeted with cynicism.” He believes that this occurs because people confuse different types of research (www.ncld.org/Research/research_types.cfm). Indeed, in the broad educational field made up of teachers, administrators, trainers, researchers, curriculum developers, tutoring providers, parents, entrepreneurs and marketers, there is not a common understanding of “scientifically based research.” This leads many educators to ask:

  • Is there an official definition of scientifically based research?
  • Is there a well-established menu of scientifically based researched programs and strategies?
  • How can we meet NCLB's research requirements?

A major challenge for educators is finding out which practices have a strong research base. Many developers of programs, practices, and textbooks say their work is “research based” because they have used some type of research to decide upon the content or approach. Such practices can only be said to be “scientifically based,” however, if data exist that show improvement for students who used the program compared to students who did not use the program. Many practices do not have this type of data because either they have not been studied (tested with students) or they have been studied, but in ways not based on rigorous, scientific methods. See the resource section for more information on specific designs that are considered rigorous.

To clarify issues surrounding research-based programs and strategies, and to serve as a foundation for resource development, the Access Center worked in conjunction with other federally supported centers, at our host organization, the American Institutes for Research. Through our work with the Center for Implementing Technology in Education (CITEd) and the National Center on Student Progress Monitoring we developed a research evidence continuum.

Level of Research Evidence Continuum

Emerging Practices Promising Practice Evidence-Based Practices
Includes practices that are not based on research or theory and on which original data have not been collected, but for which anecdotal evidence and professional wisdom exists. These include practices that practitioners have tried and feel are effective and new practices or programs that have not yet been researched. Includes practices that were developed based on theory or research, but for which an insufficient amount of original data have been collected to determine the effectiveness of the practices. Practices in this category may have been studied, but not using the most rigorous study designs. If a study uses a weak design (e.g., one-group pretest posttest) resulting evidence will be categorized as promising. If original data have been collected and a strong design has been used but the study uses a sample based on a different student population, we will note this and consider the practice promising with the special education population, but also that these practices need systematic study with students with disabilities. Includes practices for which original data have been collected to determine the effectiveness of the practice for students with disabilities. The research utilizes scientifically based rigorous research designs (i.e., randomized controlled trials, regression discontinuity designs, quasi-experiments, single subject, and qualitative research). Other less rigorous research designs may be included depending on how they compare to the Council for Exceptional Children (CEC) quality indicators.

Evidence-based practices

On one end of the spectrum are practices that have been tested empirically using rigorous research designs. Such designs show that students who use the practice benefit significantly compared to a similar group of students who do not. In this tightly controlled study, researchers are confident that the impact or change that occurred in the study is based on the practice (or intervention). Researchers can say with certainty that the change is not caused by another variable (such as student characteristics, teacher practices, or classroom conditions). Practices with a strong research base have been tested with large numbers of students and different subgroups and in a wide variety of settings. There are relatively few studies that fit these rigorous criteria, and analyzing studies to determine whether they fit this category is a complex, lengthy, and costly process.1

EXAMPLE:

A meta-analysis of research from 10 randomized-control trials focused on a particular program to calculate and synthesize effect sizes. The results show, statistically, that kindergarten and first grade students who were exposed to the program have significantly better outcomes than similar students in similar settings who did not use the program.

Promising Practices

These practices have been tested but the results are not as clear as those results in the evidenced-based research category above. Practices that fall in this category are based on some type of research – whether it is theoretical, qualitative, or quantitative – but data have yet to be collected on effectiveness. Promising practices may have been tested under different conditions and, therefore, may have a research foundation. However, the practices themselves have not been tested using the most rigorous research designs, or were tested in different educational contexts.

EXAMPLE:

A reading software program includes five major reading areas (based on the National Reading Panel’s recommendations) in its instructional model and they advertise the program as “research based.” Data from students who used the program (compared to students who did not use the program) have not yet been collected and therefore there is only theoretical and expert support for the program.

Or

An author publishes a summary of research findings from the last five years that focus on a particular policy for elementary-aged special education students. The author examined some studies that used a simple one-group pre/post-test design, and some with quasi-experimental designs. The author found that the policy studied resulted in generally positive outcomes and therefore, recommended that the policy be implemented, but noted the absence of rigorous research supporting the policy.

Emerging Practices

On the other side of the spectrum are emerging practices. These practices have been observed to be effective but do not necessarily have research to support their use.

EXAMPLE:

Four middle school teachers share what strategies have worked for them in instructing students that are diagnosed with attention-deficit disorders. They feel that these strategies help to improve student performance. The teachers decide to document the strategies and disseminate them to all teachers in the school. The teachers have their own anecdotal evidence supporting the practices, and have a lot of experience in implementing the strategies. However, the strategies have not been assessed using rigorous experimental procedures; therefore, it is difficult to determine whether the strategies used by the teachers resulted in improved student performance, or some other variable(s) impacted their performance.

Resources for Scientifically Based Research

The following are recent documents and briefs that give more information on the concept of scientifically based research and its application in education.

  • Council for Exceptional Children. Research in Special Education: Scientific Methods and Evidence-Based Practices. By: Odom, Samuel L.; Brantlinger, Ellen; Gersten , Russell ; Horner, Robert H.; Thompson, Bruce; Harris, Karen R.. Exceptional Children. Winter 2005, Vol. 71, Issue 2, 137 – 149.

This paper discusses various aspects of research-based practices and clarifies the thinking behind what is research based and what is being offered and described as research based, yet is not scientifically research based.

  • The Council for Excellence in Government. Identifying and implementing educational practices supported by rigorous evidence: A user-friendly guide. www.excelgov.org/evidence

U.S. Department of Education guide seeks to provide educational practitioners with user-friendly tools to evaluate whether an educational intervention is backed by rigorous evidence of effectiveness.

A short brief outlining and explaining Title IX’s six specific components of scientifically based research and suggestions to educators for making decisions on research in the area of comprehensive school reform.

A booklet (with accompanying audio CDs) on “the challenge and hope of scientifically based research,” components required by NCLB for Comprehensive School Reform, and ways practitioners can put scientifically based research to work.

The Clearinghouse develops systematic, detailed standards for reviewing and synthesizing educational research and presents its findings in a searchable database. The database includes reviews of potentially replicable interventions (programs, products, and practices) that are intended to enhance student outcomes. Initial topics include Interventions for Increasing K–12 Math Achievement; Peer-Assisted Learning in Elementary Schools: Reading, Mathematics, and Science Gains; Interventions for Beginning Reading; Curriculum-Based Programs for Preventing High School Dropout; Programs for Increasing Adult Literacy; Interventions to Reduce Delinquent, Disorderly, and Violent Behavior in Middle and High Schools; Interventions for Elementary School English Language Learners: Increasing English Language Acquisition and Academic Achievement; and Character Education Interventions: Benefits for Character Traits, Behavioral, and Academic Outcomes.

This document explains more about research in the educational field. It includes chapters on ‘what is research,’ ‘scientifically based research,’ ‘sources of education research,’ ‘reading education research,’ and ‘finding education research.’

T.H.E. Journal online is authoring a six-part series focused on looking at scientifically based research on education including ‘Determining what works – An interview with Dr. Grover Whitehurst,’ ‘A closer look at scientifically based research,’ and ‘Scientifically based research: Guidelines or mandates for purchasing.’

Interview focusing on the IES definition of scientifically based education research.

  • The Center for Education of the National Academies. Scientific Research in Education (Executive Summary). http://www.nap.edu/books/0309082919/html/

Discusses scientific principles and how these can be fostered in a federal education research agency.

This paper is designed to help teachers become discerning consumers of educational programs and materials. It provides guidance on how to recognize scientifically based instructional strategies, and why to use concepts of research in the classroom.

Provides a list of resources on research reviews, meta-analyses, and effect sizes,  and their applications in education.

This short overview discusses a potential framework to categorize three types of research and the connection between research and validated practice.

A digest that discusses the history of scientifically based research and its implications with regard to No Child Left Behind.

 

1 The What Works Clearinghouse was established for this purpose — to develop standards and conduct rigorous review of studies.

 

For additional information on this or other topics,
please contact The Access Center at accesscenter@air.org.
The Access Center: Improving Outcomes for All Students K-8
The Access Center is a cooperative agreement (H326K020003) funded by the U.S. Department of Education, Office of Special Education Programs, awarded to the American Institutes for Research 1000 Thomas Jefferson St. NW, Washington, DC 20007
Ph: 202-403-5000 | TTY: 877-334-3499 | Fax: 202-403-5001 |
e-mail: accesscenter@air.org website: www.k8accesscenter.org

This report was produced under U.S. Department of Education Cooperative Agreement H326K020003 with the American Institutes for Research. Jane Hauser served as the project officer. The views expressed herein do not necessarily represent the positions or policies of the Department of Education. No official endorsement by the U.S. Department of Education of any product, commodity, service or enterprise mentioned in this publication is intended or should be inferred.