An Interview with Marcus A. Winters: To Socially Promote or
Michael F. Shaughnessy
Eastern New Mexico University
Portales, New Mexico
On Wednesday, September 13, 2006, Manhattan Institute for Policy Research scholars Jay P. Greene and Marcus A. Winters released their new study on social promotion entitled "Getting Farther Ahead by Staying Behind: A Second-Year Evaluation of Florida's Policy to End Social Promotion". In this study, Greene and Winters analyze the effects of Florida's test-based promotion policy on student achievement two years after initial retention.
The issue of social promotion has been a continuing one in education for many years. There are those who believe that “retaining“ a child or having the child repeat a grade damages their self-esteem, self-concept and self worth. On the other hand, these students encounter increasingly difficult work, and frustrated teachers who realize that they have not mastered the previous years work.
Marcus A. Winters is a Senior Research Associate at the Manhattan Institute and a Doctoral Academy Fellow at the University of Arkansas. He has performed several studies on a variety of education policy issues, including high-stakes testing, charter schools, and the effects of vouchers on the public school system. He received his B.A. in political science with departmental honors from Ohio University in 2002 and his M.A. in economics from the University of Arkansas in 2006.
( Interviewer’s note : Parts of the above were taken from a Press Release as was Mr. Winter’s bio. An executive summary of this report can be found by going to: http://www.manhattan-institute.org/html/cr_49.htm
In this interview, Marcus A. Winters, one of the co-authors of the study ( along with Jay P. Green) responds to some questions about this study and the statistical analysis employed in this study as well as the ramifications and repercussions of the study.
1) You recently released a study with Jay P. Greene evaluating Florida’s policy to end social promotion. What were the basic results of this study?
We evaluated Florida’s policy that requires students to pass a standardized reading test in order to be promoted to the fourth grade. Similar policies are currently operating in Texas, New York City and Chicago. Using two research strategies, our study found that retaining these low-performing students increased their test scores compared to other students very similar to them who were socially promoted. Further, we found that the gains from retention increased in the second year, which is consistent with the idea that socially promoted students fall further behind their peers as material becomes more difficult in later grades.
2) Florida’s legislature allowed for several “exemptions” to the retention policy—students with limited English proficiency, disabled students with an I.E.P. and students who had been retained twice previously. It seems like there are a lot of “ loopholes”. Am I off base here?
There are several ways that students can be exempted from Florida’s retention policy. Other school systems with test-based promotion policies also have similar exemptions. The idea of these exemptions is to account for those students whose test score was, for some reason, an inaccurate indication of their true ability. Even if testing is a useful policy overall, there very likely are at least some outlier students whose test score does not accurately indicate their ability to read. If we can correctly identify those students who would actually benefit from retention even though they have low test scores, then these exemptions would improve the policy. However, future research of a different kind is necessary to know whether or not these exemptions have been effective or harmful.
3) Your research discusses Developmental Scale Scores. What exactly does this mean?
Development Scale Scores were developed for the state test by the Florida Department of Education to indicate a student’s absolute proficiency regardless of the grade level of the test or year in which he took it. That is, a student who earned a score of 1,000 on the 3rd grade reading test in 2004 has identical reading ability as a student who earned a score of 1,000 on the 5th grade reading test in 2002. The existence of these scores helps a great deal in our research, which compares reading proficiency of students in different grades and, in one comparison strategy, in different years as well.
4) And to give the non-statisticians a further headache, what do you mean by a regression discontinuity comparison? (Is this trying to predict something?
Regression discontinuity is a comparison strategy that is useful when a treatment is conditioned on a person reaching a particular threshold, as is the case in our study where students must reach a particular test score in order to be promoted under the policy.
The idea is to compare the performance of those who had test scores that were “just” above the threshold (and thus not retained) to the performance of students whose test score was “just” below the threshold (and thus were subject to retention). The idea is that students who fall into this narrow neighborhood of the threshold should be essentially separated by randomness. In many cases, the difference between the students is that one happened to have answered a single additional question correctly on the exam. This approach approximates a random assignment evaluation. Regression discontinuity has been growing in popularity in social science over the last several years.
5) I guess a real concern would be those “borderline kids” those who are one or two points below a certain standard or “threshold”on a certain test- how should these kids be handled?
The regression discontinuity approach helps to address this question. Using this approach, which necessarily focuses on the “borderline kids”, we find that retained students outperformed socially promoted students by about 5 percentile points after two years. These results suggest that among those whose test score was very close to that necessary for promotion, students were better off if they were retained.
6) Your study focused on Florida’s FCAT and reading. Are there other areas that should be explored or examined? Math, or spelling for instance?
Absolutely. We focus on reading because the policy depends on the reading test and the rhetorical justification for it has been to increase student literacy. However, one might also suspect that these students are also behind in other subjects so it would be interesting to evaluate the effect of the policy in those subjects as well. We do not do so here, but we may evaluate the effect of the policy in other subjects in future research.
7) Your study is somewhat disconcerting ( as are similar studies) in that we may see an initial improvement, but there is no long term follow up to see how well these kids do five, ten years down the road. Any thoughts?
Our results suggest that the effect of the policy increased from the first to the second year after retention. We might suspect that these effects will continue to compound as material becomes even more difficult in later grades but the socially promoted students still do not have the basic foundation of knowledge that they need to succeed. However, we cannot know this for sure from our study since the retained students only entered the 6th grade this year. We do, however, intend to follow these students for as long as possible in order to discover whether these gains continue to increase.
In such future work, we can also evaluate the impact of the policy on other outcomes, such as high school graduation.
8) Cost-benefit analysis and the bottom line- do you have any definitive conclusions about Florida’s policy?
It certainly is the case that ending social promotion is a relatively costly reform. The policy essentially keeps a large number of students in public schools for an additional year, which is costly given that we now spend about $10,000 per pupil in a year in public schools. However, if we find in the long term that the policy substantially increases student proficiency then it might well be worth the cost. Also, it is important to note that the policy might also improve the performance of students by giving schools an incentive to teach students better before they must be promoted. That is, even students who are not retained might benefit from the policy. Unfortunately, we cannot evaluate any such systemic effect in this paper.
9) I have to say that while some reading gains at the third level may be impressive, reading at the third grade level is different than reading at the sixth, ninth and twelfth grade level. Are we jumping to conclusions about some small percentile gains (or gains measured in standard deviations)?
The existence of Developmental Scale Scores allows us to measure absolute student proficiency. To put our results in perspective, our findings suggest that retained students have higher absolute proficiency when they enter the fifth grade than socially promoted students had when they left the fifth grade. Again, however, it is important that we continue to follow these students for as long as possible in order to discover the overall impact of the policy over the long-term.
10) What question have I neglected to ask or address?
I think that you covered quite a lot! Thank you for the opportunity to discuss this work.
