June 2 Expert Chat Transcript: Research-Based, Evidence-Based, Scientifically-Based, and Research Proven: What Does it All Mean?
The following is a transcript from an online chat event facilitated by Dr. Becki Herman and Alison Auerbach entitled “Research-Based, Evidence-Based, Scientifically-Based, and Research Proven: What Does it All Mean?” The chat, sponsored by The Access Center, took place on June 2, 2003.
ChatMaster
On behalf of the Access Center, welcome to our second expert chat event
entitled, Research-Based, Evidence-Based, Scientifically-Based,
and Research Proven: What Does it All Mean?
ALISON AUERBACH (GUEST SPEAKER)
Thanks so much for the opportunity to participate in this Access Center
Session.
ChatMaster
We are extremely fortunate to have Dr. Becki Herman, Director of the
What Works Clearinghouse facilitating this discussion. Dr. Herman is
joined by her colleague, Alison Auerbach, Communications Coordinator
for the What Works Clearinghouse. Dr. Herman and Ms. Auerbach will be
online for the next hour to answer your questions and facilitate a discussion
on this very timely topic.
ChatMaster
As the project director for the What Works Clearinghouse, Dr. Becki
Herman is responsible for the U.S. Department of Education’s flagship
project to set standards for educational research and use those standards
to identify effective educational programs, practices, and approaches.
Dr. Herman holds an M.A. and Ph.D. in sociology from Johns Hopkins University.
As Communications Coordinator for the What Works Clearinghouse, Alison
Auerbach is responsible for building and maintaining an informed network
of education consumers and producers who will contribute to and make
use of the research syntheses and on-line databases. Ms. Auerbach has
a B.A. in History from Yale University and a Masters in Elementary Education
from Teachers College, Columbia University.
ChatMaster
Who would like to pose the first question to our experts?
Judy Shanley
Hi - thanks for the opportunity to speak with you. I have occasion to
talk with lots of educators at the state and local level about the practices
they use. How can I help them use "scientifically based practices?"
Julie Duffield
What are some of the most frequently asked questions that you are asked
in this area - and how do you answer?
ALISON AUERBACH (GUEST SPEAKER)
We hope you all have had a chance to look at our website. Briefly, the
What Works Clearinghouse is a project of the U.S. Department of Education's
Institute of Education Sciences, established to provide educators, policymakers,
and the public with a central, independent, and trusted source of scientific
evidence of what works in education.
Frances Loose
The link between providing quality PD re: evidence based approaches,
implementing those approaches, and linking the PD and implementation
with student results is challenging. How will your center serve as a
resource to develop this linkage?
ALISON AUERBACH (GUEST SPEAKER)
Julie, you can read our most recent list of FAQs on the WWC website:
www.w-w-c.org.
Julie Duffield
Thanks
ChatMaster
One piece of technical advice, make sure that you do not close out of
the chat window if you are switching screens because you will then have
to log back on.
BECKI HERMAN (GUEST SPEAKER)
Judy--The Department has made a strong argument for the need for scientifically
based research by demonstrating that practitioners and policymakers
use a range of information (some very selective and rigorous, and other
far less critical) when making decisions. Influences ranging from scientific
judgment to fads to individuals’ opinions come into play. The focus
on scientifically based research demonstrates a commitment to providing
decision-makers with solid, reliable evidence so that they can make
the best decisions which will lead to improved student achievement.
How this plays out will vary at individual state, district and school
levels. In all cases, the research methods must be appropriately matched
to the question being asked. For example, if you are interested in looking
at outcomes (e.g. does a certain intervention actually result in improved
student achievement), then only certain kinds of quantitative research
are appropriate. If you are looking to answer questions about process
or the implementation
ChatMaster
I have a question for you- when can a program or strategy be called
a "promising practice" and what are the criteria?
ALISON AUERBACH (GUEST SPEAKER)
Chatmaster, there is no one right definition for a promising practice;
it depends on your perspective, what is at risk and so on. However,
the WWC is building on the standards that have been developed and used
to date and create standards which can be applied to review outcomes
research.
ALISON AUERBACH (GUEST SPEAKER)
A promising practice has some evidence --anecdotal, case studies, etc.,
-- which consistently reports effectiveness, but none of this evidence
is enough to rise to the level of saying that its effectiveness is proven.
Different criteria are used by different organizations, individuals,
clearinghouses, etc. Just as with scientific evidence, there are no
standards that are universally accepted and applied in the field of
education.
BECKI HERMAN (GUEST SPEAKER)
Frances, the What Works Clearinghouse will focus on identifying the
effects of interventions (including teaching practices) on student level
outcomes. It is entirely possible that we would look at certain types
of professional development and their impact on student learning. We
would then release an evidence report summarizing the findings from
existing research about the impact of those types of professional development.
In the first year of the WWC, we are focused especially on the effects
of interventions on student-level outcomes. Later, we might be able
to move up the causal chain. For example, if we find in the first year
that a certain teaching practice improves student learning, we might
(in the next year) look at the effects of different types of professional
development on that teaching practice.
Jane Hauser
Will the clearinghouse be examining practices across a broad range of
populations? I am thinking in particular of children with disabilities
many of whom are educated in general education settings. (Tried sending
this earlier but accidentally bumped myself out of chat.)
Katherine Goulian
I have a question: What tools or resources could you recommend to educators
at the state and local levels to help them make better decisions about
the programs and interventions they implement?
BECKI HERMAN (GUEST SPEAKER)]
Jane, yes, the clearinghouse will be looking at interventions across
a range of populations. For example, the seven topics in year one include
students in adult education, students receiving ESL instruction, students
receiving remedial early literacy instruction. In all Evidence Reports,
we will break out findings by subpopulations (e.g., students with disabilities)
where the research base is rich enough to do so. I'd be a little conservative
in extrapolating to the whole student with disability population from
such sub-analyses. We do hope to look specifically at interventions
for the target population you mentioned in a future Evidence Report.
ALISON AUERBACH (GUEST SPEAKER)
Katherine, They can turn to resources that are available to them to
find out what research or information exists on the intervention. Starting
in the fall, they will be able to go to the What Works Clearinghouse
(www.w-w-c.org). We encourage you
to look at the following resources as well for additional guidance and
information:
Institute of Education Sciences: http://www.ed.gov/offices/IES
Regional Education Labs: http://www.ed.gov/prog_info/Labs
ERIC: http://www.eric.ed.gov
The National Institute for Literacy: http://www.nifl.gov
Judy Shanley
Do you have classroom educators and SEA personnel as part of your review
process?
Don Dailey
How does the Department or the WWC respond to the notion of "craft
knowledge," or true and tried practices teachers have used that
are not studied through a study using a rigorous design. Maybe they
are too small in nature to warrant a full-scale study, but are they
worth passing on to other teaching colleagues as effective practice?
Ken Olsen
Alison or Becki - Although I haven't examined them that closely, the
definitions of Scientifically-based, research-based, or evidence-based
practice do not appear to be consistent across NCLB, IDEA and other
Federal laws. As one who works primarily with State education agency
special education units and lead state agencies for early intervention,
my concern is that practices are being promoted for which the sample
did not include students with disabilities. Will you be promoting a
definition that pushes the edge of "all" or will it be scientifically-based
for "some"?
Frances Loose
Related to Katherine's question, how can we get assistance to more effectively
document the link between interventions we implement (and PD provided)
and the impact on students within a given school year?
BECKI HERMAN (GUEST SPEAKER)
Judy, the actual reviews are conducted by trained researchers. However,
we incorporate classroom educators, SEA personnel, and other end users
of the WWC in framing the work, identifying topics, vetting products,
and sharing findings. We use vehicles such as emails, focus groups,
etc. to proactively reach out. Please see the WWC website, especially
the parts about the What Works Network.
Frances Loose
Is there a way to become involved in the WWC process?
ALISON AUERBACH (GUEST SPEAKER)
Don, a number of researchers have done good qualitative and observational
work working with teachers to document craft knowledge and best practices.
This is useful information that can be used in conjunction with outcomes
research to make informed decisions on what works in education.
ALISON AUERBACH (GUEST SPEAKER)
To continue, Don, the research base from which we are working in education
continues to evolve in much the same way as in the field of medicine.
It is always important to determine exactly what question is being asked
and then from there decide which research methods are most appropriate.
If questions being asked have to do with outcomes, then quantitative
research methods are required. Craft knowledge and other qualitative
data contribute to the shaping of a research agenda and are vital to
understanding implementation during the course of a study.
Ken Olsen
To all (re the strand on craft knowledge and qualitative data): But
haven't we been hearing clearly that only experimental studies will
qualify as evidence for scientifically research-based practice?
BECKI HERMAN (GUEST SPEAKER)
Ken, that is a good point. Part of this question is a policy question
that could be directed to the Department of Education. In terms of what
the WWC is doing, we are focusing on a narrow scientific question (effects
of educational interventions on student outcomes) and have focused our
standards for the research we will review on the types of research that
best speak to that particular question (i.e., experimental and some
kinds of quasi-experimental research). We will be looking at the types
of research conducted on interventions for students with disabilities.
As I mentioned in response to Jane's question, we also will look at
the evidence of effects of interventions on students with disabilities
wherever the research reports that.
BECKI HERMAN (GUEST SPEAKER)
I would like to mention, in case it is not clear, that the WWC will
critically review and summarize existing research, rather than conduct
original research. So the existing research base sets some parameters
for what we can report.
Trish Mengel
Will the WWC examine research in Early Intervention as well as K-12?
ALISON AUERBACH (GUEST SPEAKER)
Ken, the WWC is focused on answering a specific question --what works
in education. We are looking at research on student outcomes and this
is what will be reviewed against our standards.
BECKI HERMAN (GUEST SPEAKER)
Frances, one service the WWC will offer (starting this summer) is a
Registry of Evaluators. Individuals and organizations who conduct outcomes
research can post their contact information, qualifications, and experience
on our website. The WWC does not vouch for the quality of these evaluators'
work (only the completeness of the information they provide), but will
provide tools to help you think about how to evaluate effects and how
to identify an appropriate evaluator to help you with this activity.
ALISON AUERBACH (GUEST SPEAKER)
However, we are in no way saying that this is the only research which
should be used by education decision makers.
Frances Loose
re: craft knowledge--Given the WWC role, will craft knowledge and other
qualitative data be reflected in any way in your reporting?
Ken Olsen
Becki - We just hope that there will always be caveats about the populations
to which a practice can legitimately be generalized. If the population
was ill-defined or specifically excluded a key population, we need to
make sure everyone knows that. We don't want a repeat of the CSRD list
that generalized without population data.
ALISON AUERBACH (GUEST SPEAKER)
Trish, the WWC as identified seven topics for which it will be producing
Evidence Reports in this first year. Those topics range from interventions
for beginning reading to high school drop out prevention to adult literacy.
Frances Loose
re: Ken's question about :generalizing", With the enormous diversity
in how different states and researchers ID students w/ learning disabilities,
hence the broad array of learners included within that definition, what
is the plan to articulate as clearly as possible, for whom the results
could be expected to be replicated?
ALISON AUERBACH (GUEST SPEAKER)
Trish, any topic and population which falls within the purview of the
US Department of Education could be considered for a WWC Evidence Report.
I encourage you to visit the WWC website to find out how to nominate
a topic for future consideration.
BECKI HERMAN (GUEST SPEAKER)
Frances, you asked about craft knowledge and other qualitative data.
That type of research is very important in identifying promising interventions
that should be examined systematically. We hope that WWC users will
use this knowledge to help us identify important areas to review. However,
the work the WWC is doing, as I mentioned above, is focused on systematic
research on effects of interventions that can be generalized. See next
note...
BECKI HERMAN (GUEST SPEAKER)
Ken and Frances, you raise some key issues about generalizing findings.
We are greatly concerned with misuse of any findings, especially when
(as you point out) the samples in the research studies do not truly
represent the full population. All WWC findings will be presented with
information about the characteristics of the samples. So, for example,
we might say that intervention x improves early reading skills for disadvantaged
urban students, but that the research is silent on rural students, etc.
The research should describe "students with disabilities"
and we will present the findings as they apply to that subcategory of
students with disabilities. We will enlist the guidance of experts in
that area to ensure that we are not generalizing inappropriately.
Ken Olsen
Becki - Thank you, Thank you, Thank you! With the sub-cell AYP challenge
in NCLB, we will have to be as targeted as possible. We sure appreciate
the fact that you'll help avoid a one size fits all mentality.
Alison Auerbach (GUEST SPEAKER)
Someone asked a while ago if there is a way to become involved in the
WWC. We encourage consumers and producers of education research to become
involved and we rely on the input of practitioners, researchers, policymakers
and others as we move forward in our work.
ChatMaster
It is time to conclude today’s chat event. We are so pleased about the
participation and the level of discourse. Our sincere thanks to both
Dr. Becki Herman and Alison Auerbach for sharing their experience and
expertise
ChatMaster
You can continue this dialogue as a threaded discussion on the Access
Center website’s Interactive Online section. Simply go to this website:
http://www.k8accesscenter.org/online_
community_area/ threaded_discussion.asp
Frances Loose
Thank you for the opportunity to participate in the chat. Now that I've
participated, I'll know how to better prepare for the next one.
ChatMaster
A transcript of this chat will be available shortly on the Access Center
website at: www.k8accesscenter.org Please feel free
to share it with your colleagues
ChatMaster
We would appreciate your feedback on the chat event today. Please take
a minute to fill out a brief, anonymous evaluation form. Click on the
following link for the evaluation form: http://www.k8accesscenter.org/feedback/
default.asp
Alison Auerbach (GUEST SPEAKER)
Thank you for this opportunity. We welcome additional questions about
the WWC at wwcinfo@w-w-c.org.
ChatMaster
Watch for the date announcement for the next Access Center expert chat
event in your email and on our website: www.k8accesscenter.org.
ChatMaster
Goodbye folks!
