-
IES Holds its First Workshop on How States and Districts Can Evaluate Their Interventions
May 1, 2008
Empirical Education staff members participated in a workshop held in Washington sponsored by the US Department of Education’s Institute of Education Sciences (IES) on April 24. The goal of the event was to encourage “locally initiated impact evaluations.” The primary presenter, Mark Lipsey of Vanderbilt University, called upon his extensive experience in rigorous evaluations to illustrate and explain the how and why of local evaluations. David Holdzkom of Wake County (NC) provided the perspective from a district where the staff has been conducting rigorous research on their own programs. Empirical Education assisted IES in preparing for the workshop by helping to recruit school district participants and presenters. Of the approximately 150 participants, 40% represented state and local education agencies (the other 60% were from universities, colleges, private agencies, R&D groups, and research companies). An underlying rationale for this workshop was the release of an RFP on this topic (see our synopsis). Empirical Education expects to be working with a number of school districts on their responses to this solicitation, which is due October 2, 2008.
-
Two-Year Study on Effectiveness of Graphing Calculators Released
March 1, 2008
Results are in from a two-year randomized control trial on the effectiveness of graphing calculators on Algebra and Geometry achievement. Two reports are now available for this project, which was sponsored by Texas Instruments. In year 1 we contrasted business as usual in the math classes of two California school districts with classes equipped with sets of graphing calculators and led by teachers who received training in their use. In the second year we contrasted calculator-only classrooms with those also equipped with a calculator-based wireless networking system. The project tracked achievement through state and other standardized test scores and implementation through surveys and observations. For the most part, the experiment could not discern an impact as a result of providing the equipment and training for the teachers. Data from surveys and observations make clear that the technology was not used extensively (and by some, not at all) suggesting that training, usability, and alignment issues must be addressed in adoption of this kind of program. There were modest effects, especially for Geometry, but these were often not found consistently for the two measurement scales. In one case contradictory results for the two school districts suggests that researchers should use caution in combining data from different settings.
-
At NCES Conference, Empirical Education Explains a Difficulty in Linking Student and Teacher Records
February 29, 2008
The San Francisco Bay Area, our “home town”, was the site for the 2008 National Center for Educational Statistics (NCES) conference “Data by the Bay.” From February 25 to 29, educators and researchers from all over the country came to discuss data collection and analysis. Laurel Sterling and Robert Smith of Empirical Education presented “Tracking Teachers of Instruction for Data Accuracy and Improving Educational Outcomes”. Their topic was the need to differentiate between, the teachers who actually perform instruction for students and the teachers with whom those students are officially registered. They explained that in our research we keep track of the “teacher of instruction” vs. “teacher of registration”. Without this distinction we are unable to properly identify the student clusters or associate student growth with the right teacher. Sharing instructional responsibilities within a grade-level team is common enough to be of concern. In a large experiment involving teachers in grades 4 through 8, 17% reported teaching students who were not assigned to them on the official class roster. The audience was very lively and in the question period contributed to the topic. One district IT manager indicated that there is movement in this direction even at the state level. For a copy of the presentation, send an email to Laurel Sterling.
-
Empirical at 2008 AERA Conference
December 2007
Acceptance notifications are in! Empirical Education will have a strong showing at the 2008 American Educational Research Association conference, which will be held in New York on March 24–28, 2008. Empirical staff will be presenting their research and findings in several divisions and special interest groups, including School Evaluation and Program Development, Learning and Instruction, Survey Research in Education, and Measurement and Research Methodology: Quantitative Methods and Statistical Theory. Our six presentations will include a variety of topics:
- Evaluating Alabama’s Math, Science, and Technology Initiative: Initial Results of a State-Wide Randomized Experiment
- Matched-Pairs Designs and Standard Errors of Impact Estimates: Lessons from Ten Experiments
- Effects of Pairing on the Precision of Moderator Effects: Lessons from Nine Cluster Randomized Trials
- Comparative Effectiveness of the TI-84 Graphing Calculator on Geometry Outcomes
- Multi-Site Randomized Field Trial of an Elementary Science Curriculum
- Measuring Time and Implementation in the Alabama Mathematics, Science, and Technology Initiative
View pictures of Empirical Education’s reception at The Chocolate Bar in NYC.
-
Empirical Education Pilots Workshop at REL-NEI Regional Meeting
November 14, 2007
Providence, RI was the site of REL Northeast and Islands’ 2007 Regional Meeting on Teacher Quality. Empirical Education staff headed out to the east coast to pilot the “Becoming Good Consumers of Research” workshop to an audience of about 30 education researchers, school administrators, university professors, and other education professionals interested in using research to inform school decisions. Gloria Miller, the company’s director of evaluation design, facilitated group discussions that touched on identifying different types of research and potential sources of bias. Gloria also provided participants with a Critical Reader:rsquo;s toolkit designed to help readers evaluate the trustworthiness and relevance of various pieces of research. “Consumers of Research” is the first in a planned series of workshops focused on increasing the understanding and use of research in schools. The next round of workshops, scheduled for March 2008, is currently being developed.
-
Report Released on the Effectiveness of Carnegie Learning’s Bridge to Algebra
October 15, 2007
Empirical Education released the results of a year-long randomized experiment on the comparative effectiveness of Carnegie Learning’s Bridge to Algebra program. This study, like that on Carnegie Learning’s Cognitive Tutor for Algebra (see news for May 24, 2007), was conducted in cooperation with the Maui School District in Hawai’i and funded through a grant to Empirical Education from the U.S. Department of Education. The experiment could not discern an overall difference between the program and control group measured by either NWEA’s test for general math or the test’s algebraic operations subscale. However, for the algebraic operations outcomes (but not for general math), we found that students scoring low before participating in Bridge to Algebra benefitted significantly more from the program’s algebraic operations instruction than did students with high initial scores. The district was specifically interested in looking at how the different ethnic groups, particularly the Hawaiian/Part-Hawaiian and Filipino students, performed in the new program. Controlling for pretest, we did not find that the program had a different effect for different ethnicities. The district was also interested in learning whether the program was differentially effective for students taught by certified teachers versus those with non–certified teachers. For the overall score (but not the algebraic operations sub-strand), we found that the program gave the non–certified teachers an advantage. Finally, despite some implementation challenges, teachers reported a generally positive attitude toward the new program.
-
Major Study of Elementary Science Reveals Reading Improvement
August 3, 2007
Empirical Education released the report of a randomized control trial of Scott Foresman Science. The report concludes that the program shows promise as an enhancement of the school’s reading program. The study encompassed five school districts in five different states and included more than 80 third- through fifth-grade teachers divided randomly between those who were trained and provided with the science text and materials and those who continued with their existing science materials. The Scott Foresman Science materials were geared to reading, providing leveled readers corresponding to each chapter of the text. The study was sponsored by Pearson Education.
-
Empirical Education Presents at Annual IES Conference
June 7, 2007
Empirical Education staff members were part of two presentations at the Institute of Education Sciences (IES) Research Conference in Washington D.C. Xin Wei and Denis Newman presented a study on paired randomization, a technique that the company uses extensively to improve the design of local school district experiments. The research, led by Andrew Jaciw, the company's director of research design and analysis, drew on data from 12 experiments and provided an analysis of the statistical advantages of the approach. A full report is being prepared for presentation at AERA. The work is supported by a grant from the IES. A second presentation provided a preview of the state-wide experiment evaluating the effectiveness of the Alabama Math, Science and Technology Initiative. Empirical Education will begin analyzing the outcomes for the first year in fall 2007. Denis Newman and his co-PI, Richard Sawyer of Academy for Educational Development, led the presentation. The work is part of the Regional Education Lab in the Southeast.
-
Experiment Findings about Cognitive Tutor Not What Was Expected
May 24, 2007
Empirical Education released the results of its year-long randomized experiment on the comparative effectiveness of the Cognitive Tutor program, conducted in cooperation with the Maui School District in Hawai’i. Funded through a grant from the US Department of Education to Empirical, the research evaluated the local implementation of Carnegie Learning’s Algebra I product. Company president Denis Newman explained the results, saying, “We were surprised by the results and consider this study something we can all learn from. Nobody expected a negative impact on the experienced teachers but this may show that it takes some time to get back up to their high level of performance when learning a new way of teaching. Following these teachers’ progress into a second year would help to verify this explanation. This effect was not found in previous research, which again points to the importance of conducting many experiments on any intervention before coming to a conclusion about its effectiveness.” A second experiment in the same school system, focused on the new Bridge to Algebra product, is nearing completion.
-
Empirical Education Reports Research at AERA
April 12, 2007
Staff and collaborators in Empirical Education research presented their findings this month at the American Educational Research Association’s annual conference—the primary meeting of the education research community. The company’s research and expertise was highlighted in
six places on the program including contributions in the divisions for Learning & Instruction, Policy & Politics, Measurement & Research Methodology, and School Evaluation & Program Development.