Responses to: “Data-Driven Decision Making—Applications at the District Level” April 2008
I think products meant for SBR should be made available to teachers and administrators who are directly involved with the students to test before being finally approved for general use.
Teachers should be selected from school systems to participate in material selection for use in classroom and science laboratories.
Education as we know is not a plastic making indusry at the mercy of producers. We deal with human quality production. Therefore those of us in the classroom should be part of the material selection process and not lobbyist and those working for their prospective gains.
— Simeon Bomi, Lowndes County Public Schools
(Return to this latest evidence piece)
(Respond to this opinion piece)
What’s Unfair about a Margin of Error? January 2008
Margin of error is also a problem when using test data to track individual students into groups based on skill level. Low test precision will result in many students being placed in the wrong group:
http://www.geocities.com/gordonite32/misc/testsupport.htm#stddev
— Karl, teacher
(Return to this latest evidence piece)
(Respond to this opinion piece)
Responses to: “What Happens When a Publisher Doesn’t Have Scientific Evidence?” December 2007
Bravo, for a thoughtful analysis of the CREW complaints! I heartily endorse the call for evidence of effectiveness of curriculum products, but I would add two cautions to interpretation of this evidence by schools:
1) evidence of effectiveness is always a highly conditional finding. Even with fairly large-scale, multi-million dollar RCT studies, we have seen that there are powerful interactions with school effects, student effects, teacher effects, and other conditions of implementation. It is often the case that the range of effects is from strongly positive to strongly negative, even if the mean effect is positive. Thus, it is very unlikely that we will ever have conclusive, scientifically based evidence of effectiveness which would be so strong and unambiguous as to become a prerequisite for eligibility for funding. Schools should be required to take into account evidence of effectiveness in their purchase decisions, but they should also have the latitude to interpret the applicability of the evidence to their context.
2) in the present climate, it is very difficult for a small supplemental curriculum product to demonstrate a meaningful effect size on a measure which is meaningful to policy makers, such as a state test. Often these products are used for only a few minutes per week, or for a small portion of a school year, and the tests are not sensitive enough to show the impact of any intervention which is so small. Also, these products often do not have a restricted set of prescribed uses, so implementation varies widely. Effects are far from uniform, and are very difficult to measure directly with conventional achievement tests.
These objections are not against the principle of the need for evidence of effectiveness, but they do point out practical limitations. I would hope that implementation of NCLB reauthorization would take these limitations into effect.
— Rob Foshay, Director of Research, Texas Instruments
(Return to this latest evidence piece)
(Respond to this opinion piece)
Responses to: “Congress Grapples with the Meaning of ‘Scientific Research’” October 2007
Great argument!
One thing you could have made explicit: the “approved list” issue comes not from the law, but from its implementation. In the case of Reading First, the administrative regulations led to creation of the “approved list” -- with the famous abuses. In the case of other products and curriculum areas, the problem is that it’s not reasonable for schools to engage in comprehensive reviews of research to find the right product, so the “approved list” practice simply becomes a de facto way means of compliance.
The question I have is whether it’s realistic to expect schools to do evaluaton research which meets any plausible definition of technical competence beyond “action research.” After all, the schools’ concern is with effectiveness of their overall programs of curriculum and instruction; it would be unlikely that even a district large enough to do it would care about isolating the effects of a particular product, nor would they do a study designed for that purpose. Much more likely is that the schools will simply examine the end-of-period test results to decide how well their curriculum is working. If they are interested in the effects of an individual product within that curriculum, they’ll simply ask their teachers whether the product was part of the solution or part of the problem. If you don’t care about generalizability, and you do realize how many interaction effects govern what actually happens in any given classroom, then that’s probably a more informative methodology than an RCT study.
Except in very unusual circumstances, I’ve yet to see a school district feel the need to go beyond a correlational or quasi-experimental design -- and those, usually done badly. And again, the unit of analysis is usually comparison of the whole year’s program to last year’s result (or maybe results in another school using a different program), with no quantitative attempt to handle differences between the treatment and control groups. So there is no possibility of isolating the effects of a particular product.
Bottom line: there’s certainly nothing wrong with giving schools the option of doing their own evaluations as an alternative to the “approved list” -- but I doubt if it will have much effect on improving use of sound scientific evidence to make decisions in schools.
— Rob Foshay, Director of Research, Texas Instruments
What Dr. Foshay noted about research and evaluation in schools is indeed a great challenge to turn educators from consumers to producers of evidence. That said, some high achieving and affluent districts have taken the initiative to hire Ph.D.s to conduct rigorous research in local setting (e.g., Montgomery School District in Maryland). Institutions like Empirical Education can also help by conducting evaluation studies for local schools as long as data warehouse and funding are at place. Too much money has been spent on expensive large scale studies that have values for policy making at the federal level but little use for local schools. More money should be given schools and districts to hire high quality researchers or contract out much less expensive local evaluation projects to research institutions.
On the other side, training can be provided to existing school-based researchers who are not equipped with the knowledge and skills to conduct scientific research. With an emphasis on practice, it should not be too difficult to train them how to do T-test or ANOVA based on random samples or use regression or ANCOVA to control for confounding factors. Moreover, research institutions can safeguard the validity of the locally produced evidence by reviewing the local studies and provoding consulting services regarding research design, data analysis, and results interpretation.
— Bo Yan, Ph.D., Program Evaluator, Blue Valley School District
(Return to this latest evidence piece)
(Respond to this opinion piece)