February 24 Expert Chat Transcript: Large Scale Assessment
The following is a transcript from an online chat event facilitated
by Mr. Michael Hock entitled “Accountability Assessment and Students
with Disabilities: Myths and Realities .” The chat, sponsored by The
Access Center, took place on February 24th, 2004.
ChatMaster
On behalf of the Access Center, welcome to our
expert online chat event.
ChatMaster
We have experienced some technical difficulties with our chat
software this morning, but we are hopeful that everything will run smoothly
during our session today. If you do experience problems and get disconnected,
please call our technical support at 202-295-6906 or check our homepage
www.k8accesscenter.org for updates on the chat status.
ChatMaster
Back to the matter at hand: today's topic is "Accountability Assessment and Students with Disabilities: Myths and Realities."
ChatMaster
We are extremely fortunate to have Mr. Michael Hock facilitating this
discussion.
ChatMaster
Michael Hock is a Program Associate at WestEd's Northeast Regional
Resource Center (NERRC) where he leads an LD Identification Initiative
with a focus on Alternate and Large Scale Assessment.
ChatMaster
Prior to joining WestEd, Michael was a special educator for
over thirty years. He continues to teach courses on special education
assessment at the University of Vermont. For the past eight years he
has provided his technical expertise to the Vermont Department of Education
on issues such as leadership and the state's alternate assessment project.
ChatMaster
Michael will be online for the next hour to answer your questions and
facilitate a discussion on this very timely topic.
ChatMaster
Once again, if you experience any technical difficulties during the
chat, please feel free to call our technical help line at 202-295-6906.
ChatMaster
Let the chatting begin!
Michael Hock
First of all, I want to offer some shorthand to help make the “digichat”
process easier:
SPED is special education
GENED is general education
SWDs are students with disabilities
GENEDs are general education students
ELLs are English Language Learners
IDEA is the Individuals with Disabilities Education Act
NCLB is the No Child Left Behind Act
AYP is adequate yearly progress
ED is the US Office of Education
OESE is the US Office of Elementary and Secondary Education
OSEP is the US Office of Special Education Programs
SEAs are State Education Agencies
LEAs are Local Education Agencies
Michael Hock
Anyone have any additional shorthand to make this easier?
Michael Hock
By the way, this is a first digichat for me. It was suggested that I prep some things beforehand and cut and paste. The paste I just did is a little weird.
ChatMaster
I think everyone would agree with those terms. It will certainly make things easier.
Kristin Reedy
This is the first one for me, too. What do we do?
ChatMaster
Michael will share information with everyone and participants can ask him questions as well
Michael Hock
Well let's try this. I have some opening ideas to prompt some discussion.
Michael Hock
To get the conversation rolling, I thought we might explore some of the myths and realities related to the notion of an achievement gap between GENED and SPED students. A good way to do this is to look at a study by Bielinski and Ysseldyke called “Interpreting Trends in the Performance of Special Education Students” published in 2000. If you're interested, you can find this study in the National Center on Educational Outcomes' website. Stick with me. This is kind of long, but it will be great context for our discussion. First, some background assumptions: A major cornerstone of NCLB is adequate yearly progress (AYP), which assumes that annual assessment, coupled with a system of rewards and sanctions, will lead to yearly gains in student achievement that will eventually add up to ALL students meeting academic achievement standards. A major component of the AYP logic model is that accountability assessment will be sensitive to improvements in curriculum, instruction, supports and services.
Michael Hock
The idea is that schools will use assessment results to target program improvement and overtime the success of those improvements will show up in assessment results. NCLB requires states and local educational agencies to disaggregate overall assessment results into four cohorts groups (SWDs, ELLs, Low SES Status, and major racial/ethnic groups) to assure that accountability assessment and reporting will demonstrate the success or failure of the range of school programs and services. One major component of the AYP logic model is that cohort membership will remain relatively constant so that the cohort group that should have benefited from the effects of systems change and program improvement will include the students who were the direct beneficiaries of those changes. In other words, the model doesn't work very well if success is a ticket out of the cohort group.
Michael Hock
So, this all means that states and schools must be accountable for all students, including SWDs, and that their success (or lack thereof) in providing effective programs will show up in test scores. With those assumptions in mind, here are some highlights from the Bielinski and Ysseldyke study:
Kristin Reedy
What do you think about the new guidance/flexibility with regard to ELL Students? Could the same logic be applied to SWDs?
Michael Hock
The study was designed to help answer the question: Can disaggregated test results for SWDs be used to make valid inferences about whether students with disabilities are benefiting from education to the same degree as their non-disabled peers
Michael Hock
Re: Kristin's question: as we continue to look at this study, we may conclude that this sort of flexibility will be needed. Again, the AYP model only works if the assessments are sensitive to changes in programs that ultimately produce positive results for kids.
Michael Hock
Back to Beilinski & Ysseldyke: the study was designed to help answer the question: Can disaggregated test results for SWDs be used to make valid inferences about whether students with disabilities are benefiting from education to the same degree as their non-disabled peers
Michael Hock
They looked at a large longitudinal database of test scores for SPED and GENED students that covered over 6 years and included almost 30 thousand SWDs. They found that there was an average yearly turnover of about 20% in the SWD cohort. Actual turnover was greater in the lower grades and declined toward grade 8. The turnover was bi-directional. The lowest performing students in GENED were moved to SPED, and the highest performing SPED students were moved to GENED. The result was an achievement gap between GENED and SPED cohorts that got wider every year. When results for individual students were analyzed, the gap actually decreased slightly.
ChatMaster
Just in case someone logged in late and missed it: SWD is: students with disabilities
Michael Hock
Myths and Realities? The Achievement Gap between GENED and SPED cohorts appears to be real. It was evident in this study and it is emerging as a major factor in many states' accountability reporting. Ed Week's recent “Quality Counts” report showed a gap of 30 or more percentage points in 75-90 percent (depending on grade level) of the 39 states that provided them with data. It's not the only gap we need to address, however. There also appears to be a huge gap related to SES status.
Michael Hock
The question is whether or not the gap is evidence of a failing system, or simply a problem of inaccurate inferences…or both!
Michael Hock
The myth is that we might as well ignore the achievement gap because there is nothing we can do about it.
Michael Hock
What are your thoughts? Any questions or comments this brings up for you?
Cybele Werts
Could you explain more about what you mean by inaccurate inferences?
Michael Hock
Yes, Cybele. The "inaccurate inference" we would be concerned about in accountability assessment would be to decide that programs were failing when, in fact, the assessments might not accurately reflect changes or functions of curriculum or assessment or services
Byron Rice
This process is new to me as well. My question is: Does the current assessment process truly measure the academic growth of the bulk of our sped population? The majority of our sped population does not fall into the high range of academic ability and thus will struggle on state assessments. We have very few students who qualify for the Alternate Assessment. Yet the bulk of our sped students are expected to perform at the same proficiency level as is every other general ed student. Some point out that these kids perform below proficiency levels in key academic areas and that is exactly why they are in sped. Many of these students cannot read at their grade level and as a result their performance on a standardized assessment will be skewed. Others state that placing such an assessment in front of a child (at the child's grade level) who cannot even read the front page in unforgivable. Some state that for this population of students, the current assessment process does not result in an accurate picture
Michael Hock
That was the concern in the study I sighted. That the SWD cohort was too much of a moving target to show if programs were working because the successes of special education were moving back to GENED and the lowest performers in GENED were moving into special education.
Amanda Schwartz
What are the implications of accountability and subgrouping on the referral, identification, and placement of CWD?
Elaine Bonner-Tompkins
Evidence from outlier schools, schools that have a small or no gap between students based on special education status, would suggest that there is much that can be done to close the achievement gap. Does anyone have any information re: what states are doing here? Are they identifying and studying these schools? I know some work has been done to identify those schools that close the gap by ethnicity and income. I am less familiar with schools that have done this based on disability or English language proficiency
Michael Hock
Amanda, your question is really interesting. I was just reading an article in "The special Educator" suggesting that we need to revisit the Rowley Decision in light of the new "standards movement."
Elaine Bonner-Tompkins
I am not familiar with the Rowley decision
Michael Hock
If you recall, Rowley was the first special education case to go to the Supreme Court. It offered a two-prong test for determining if SWDs were receiving a free appropriate public education...that due process was followed and that their programs were "reasonably calculated to provide educational benefit
Kristin Reedy
If we hypothesize that there are many children in special education now who really don't have disabilities but are underachieving as a result of poor/inadequate instruction, theoretically those students should be able to achieve proficiency and get out of special ed. (if they do receive adequate instruction.) Similarly, if we provide good instruction to begin with, these students will never be in special ed. The remaining students in special ed. would be less likely to achieve proficiency and the gap between general ed. and special ed. students would get wider. Ironically, in a way, the widening of the gap would be evidence that our programs are effective and that we are identifying the "right" students for special education.
Michael Hock
Thanks Kristin. You've brought us back to the myth...that there is no hope of making AYP for students with disabilities
Michael Hock
And Elaine, we are doing a lot to improve programs for these students. That's why there is an access center and the "What Works Clearinghouse" and folks helping us learn about universal design for instruction
Elizabeth Smith
I disagree Michael. If students were assessed on the level at which instruction is occurring, (the developmentally appropriate place) instead of tying it directly to grade level, I think that there would be a better chance at achieving AYP. In that way a 6th grade SED student who is receiving instruction for 3rd grade, would be assessed at the third, rather than 6th grade level.
Amanda Schwartz
The onus is then on the IEP committee for making appropriate plans and the teaching staff to implement them. How do we educate staff to understand and implement this? How can resources like these be implemented in districts and schools?
Barbara Thrush
How would we then make comparisons from year to year if we test students out of level?
Michael Hock
I need a second to think about and respond to Elizabeth
Kristin Reedy
Elizabeth, that sounds like redefining AYP based upon the student's rate of progress, his performance compared to himself, not others.
Elizabeth Smith
If we are assessing levels Kristen, he or she would be assessed in relationship to that third grade cohort. And Barbara, I don't have a magic answer for that one - but it is a place that I have been wrestling with for some time.
Michael Hock
Elizabeth, I'm hesitating because I generally agree with you...assessments should show what students know, not just what they don't know.
Byron Rice
Components of state assessments and NCLB are supposed to be designed to measure student growth and effective instruction which in turn determine AYP - my concern is that in most districts in our state did not meet AYP in part, based of the performance of SWD. Is the current assessment process equitable? What is the solution - out of level testing?
Elaine Bonner-Tompkins
Not to interpret Elizabeth's concern, but I think the larger issue is the need for AYP to reflect gains in student progress on meeting state standards. Through the value-added approach, comparisons of progress could be made across groups.
Michael Hock
However, it creates a huge test development problem. How do we include enough items to capture all the skills represented by the students at a particular grade level, without making many kids take overly long tests?
Elizabeth Smith
I am not sure what you are asking. Assessments at the state level are already designed to assess students at a single grade level based on adopted Academic Content Standards.
Michael Hock
Elaine and Byron make great points...but the measure of accountability in NCLB is not individual student progress but the percentage of students who are meeting academic achievement standards at grade level
Aran Felix
Can you define value-added approach?
Michael Hock
Elaine, I think that is a question for you, not me...
Elaine Bonner-Tompkins
I am no expert, but understand that systems can be developed to monitor individual student progress longitudinally.
Elaine Bonner-Tompkins
A few states (TN for example) have developed systems where they can account for how students, grade by grade, meet state standards. You can monitor teacher quality in that fashion, among other items.
Elaine Bonner-Tompkins
So, instead of comparing this year's third grades to last year's, you can measure progress among individual sets of students.
Michael Hock
They definitely can. At least one amendment to NCLB has been proposed that would change accountability for SWDs to a progress model. But it never went anywhere. The current model is performance against grade level standards, not the amount of progress made.
Elaine Bonner-Tompkins
If any one else would like to chime in on this, please do. I will share a reference for more info shortly.
Susan Taylor
Value-added scores are part of the state's Tennessee Comprehensive Assessment Program, or TCAP, which was started in 1992. Every year, students in grades 3-8 take standardized tests in reading, language arts, math, science and social studies. Those tests produce scores in two categories: achievement scores, which show how much students know compared with their peers across the nation; and value-added scores, which show whether students learned more or less than they should have during the school year.
Susan Taylor
Those tests produce scores in two categories: achievement scores, which show how much students know compared with their peers across the nation; and value-added scores, which show whether students learned more or less than they should have during the school year.
Michael Hock
So, we've talked a lot about ways we can make assessments more accurate and equitable. But what about eliminating the achievement gap in the classroom. What can we do about that?
Michael Hock
One thing I would really like to see is some sort of universal screening or curriculum-based progress monitoring in every
Michael Hock
General education classroom. Special education should not be a "wait-to-fail" situation.
Amanda Schwartz
What are your thoughts on portfolio assessment or the "Work Sampling System"?
Chuck Hitchcock
We need to examine the barriers that are inherent within the general education curriculum (methods, materials and assessments) then consider the development, acquisition and implementation of a curriculum designed to support all learners, not just a few. That's one part of Universal Design for Learning.
Michael Hock
I think it's great. We should not sacrifice authenticity for the sake of expediency. I would like to see multiple measures used in accountability...a combination of on-demand tests, as well as deeper, more authentic local assessments.
Michael Hock
Thanks, Chuck. I certainly agree. We probably wouldn't be having this discussion if classrooms were applying universal design...or at least the discussion would be about a much smaller group of kids
Chuck Hitchcock
As you probably know, a significant amount of energy and sped resources are devoted to "fixing up" approaches and materials. This, as you well know, applies to local and large-scale assessments as well.
Michael Hock
I agree that we need to improve assessments. But sometimes the discussion about the problem with assessment diverts us from the real problem...that our special services aren't working for a lot of kids...and that general education is not addressing the diversity of skills that are present in those classrooms
Ina Woolman
In Rhode Island we have instituted a Personal Literacy Program requirement for any elementary student reading below grade level - many schools are extending this to other curriculum areas as well, and for all students. It's a system of attention to student need right away, with interventions designed to bring the student up in the near future. And it's based on a foundation of a comprehensive early literacy program. We expect this to reduce the numbers of students who end up being referred for special education evaluation. Differentiation of instruction is an integral part of the picture. I should add thanks to Heartland AEA in Iowa for inspiring this direction, and to Michael for ongoing help with this "response to intervention" approach.
Chuck Hitchcock
We have devoted a lot of effort to improving curriculum materials (textbooks, trade books) and software so that we will have modes for publishers to use when developing the resources that teachers need to support all learners. This is just a foundation for a universally designed curriculum - just a start.
Elaine Bonner-Tompkins
I need to log off, but wanted to share a resource on value added approaches - ECS has a recent report on Improving Academic Achievement in Urban Districts that has a chapter on it (chapter 4). www.ecs.org
Michael Hock
Thanks Elaine. I don't want to leave this discussion without making what I think is one critical point about accountability assessment. We need to provide classroom teachers with professional development on analyzing assessment results and then we need to include them in the process of analyzing the test data. When analyzing assessment results, don't assume that low scores in math or reading necessarily mean that there is something wrong with your GENED curriculum. It might be some other problem that has reared its head in your math scores. It might be that you have a group of students who need to have breakfast before going into the classroom. That's why we need to have teachers involved in analyzing accountability results. We often need to link the numbers to real kids to avoid the MYTHS.
Susan Taylor
Thanks Michael and all. This has been very helpful.
Michael Hock
That gets back to the idea of inaccurate inference. Our statewide tests might just be a "vital sign" of the health of our programs. We need to dig into other data sources to find out what's really going on. And we need a multi-disciplinary approach to data analysis and action planning
ChatMaster
It is time to conclude today's chat event. Our sincere thanks to Mr. Michael Hock for participating in this event. We are so pleased with the participation and the level of discourse of all participants.
ChatMaster
The Access Center is actively working to improve the structure of our chat events and we need your help. Please take a minute to fill out a brief, anonymous evaluation form. Click on the following link http://www.k8accesscenter.org/feedback/. Please include any helpful hints you have about what we can do to make these sessions even more successful by including your specific feedback in the "please provide any additional comments" section of the form. You may also email us at accesscenter@air.org with any additional feedback.
Michael Hock
thank you chat master...
ChatMaster
As a follow-up to this chat event, we encourage you to use our listserv to continue conversing about large-scale assessment and other issues relating to access. If you have not already signed up for our free listserv, simply visit http://www.k8accesscenter.org/listserve.asp and provide your email address. We will also be sending announcements about our future chats, including topics such as Learning Strategies.
ChatMaster
You can also participate in our threaded discussion sessions by going to http://www.k8accesscenter.org/discuss/discuss.asp?mode=room&catID=182
ChatMaster
Additionally, a transcript and summary of this chat will be available on the Access Center website at: www.k8accesscenter.org ...
ChatMaster
Watch for announcements about future events on our listserv and our website. We look forward to chatting again!
ChatMaster
Thank you everyone. Have a great afternoon!
