-
REL West Releases Report of RCT on Problem-Based Economics Conducted with Empirical Ed Help
July 30, 2010
Three years ago, Empirical Education began assisting the Regional Educational Laboratory West (REL West) housed at WestEd in conducting a large-scale randomized experiment on the effectiveness of the Problem-Based Economics (PBE) curriculum.
Today, the Institute of Education Sciences released the final report indicating a significant impact of the program for students in 12th grade as measured by the Test of Economic Literacy. In addition to the primary focus on student achievement outcomes, the study examined changes in teachers’ content knowledge in economics, their pedagogical practices, and satisfaction with the curriculum. The report, Effects of Problem Based Economics on High School Economics Instruction is found on the IES website.
Eighty Arizona and California school districts participated in the study, which encompassed 84 teachers and over 8,000 students. Empirical Education was responsible for major aspects of research operations, which involved collecting, tracking, scoring, and warehousing all data including rosters and student records from the districts, as well as the distribution of the PBE curricular materials, assessments, and student and teacher surveys. To handle the high volume and multiple administrations of surveys and assessments, we created a detail-oriented operation including schedules for following up with survey responses where we achieved response rates of over 95% for both teacher and student surveys. The experienced team of research managers, RAs and data warehouse engineers maintained a rigorous 3-day turnaround for gathering end-of-unit exams and sending score reports to each teacher. The complete, documented dataset was delivered to the researchers at WestEd as our contribution to this REL West achievement.
-
AERA 2010 Recap
May 20, 2010
Empirical Education had a strong showing at the American Educational Research Association’s annual conference this year in Denver, Colorado. Copies of our poster and paper presentations are available for download by clicking the links below. We also enjoyed seeing so many of you at our reception at Cru Wine Bar. View the picture collage from our event!
Formative and Summative Evaluations of Math Interventions, Paper Discussion
Division: Division H - Research, Evaluation and Assessment in Schools
Section 2: Program Evaluation in School Settings
Chair: Dale Whittington (Shaker Heights City School District)
Measuring the Impact of a Math Program as It Is Rolled Out Over Several YearsReading, Written Expression, and Language Arts, Poster Session
Division: Division C - Learning and Instruction
Section 1: Reading, Writing, and Language Arts
Examining the Efficacy of a Sight-Word Reading Program for Students With Significant Cognitive Disabilities: Phase 2Statistical Theory and Quantitative Methods, Poster Session
Division: Division D - Measurement and Research Methodology
Section 2: Quantitative Methods and Statistical Theory
Matched Pairs, ICCs, and R-Squared: Lessons From Several Effectiveness Trials in EducationFormative Evaluations of Educational Programs, Poster Session
Division: Division H - Research, Evaluation and Assessment in Schools
Section 2: Program Evaluation in School Settings
Addressing Challenges of Within-School Randomization -
Software Industry Sets High Standards for Product Evaluation Research
May 13, 2010
The Software & Information Industry Association (SIIA) announced the release of their new report, “Conducting and Reporting Product Evaluation Research: Guidelines and Considerations for Educational Technology Publishers and Developers.” Authored by our very own Dr. Denis Newman under the direction of the SIIA Education Division’s Research & Evaluation Working Group, the guidelines provide practical considerations and share best practices of product evaluation design, conduct, and reporting. Written primarily for publishers and developers of education technology, the guidelines reflect the high standards necessary to carry out rigorous, unbiased effectiveness research. Reviewers of the guidelines included Larry Hedges with Northwestern University, Robert Slavin with Johns Hopkins University, and Talbot Bielefeldt with the International Society for Technology in Education (ISTE). A delegation of software publishers presented the Guidelines May 17 at the US Department of Education to John Q. Easton (Director of IES) and Karen Cator (Director of the Office of Education Technology). The document is now available to the public at the link above.
-
Report Released on Phase Two of The Efficacy of PCI’s Reading Program
April 14, 2010
The results are in for Phase Two of a five year longitudinal efficacy trial of PCI’s Reading Program for students with moderate to severe disabilities. This research builds upon an initial randomized control trial conducted last year that found that students in the PCI program had substantial success in learning sight words in comparison to students in the control group. Phase Two continues research in the Brevard and Miami–Dade County school districts with teachers of supported-level students in grades 3-8. Using both quasi-experimental and extra-experimental methods, findings again demonstrate that students who received PCI for two years achieved significantly higher scores on the sight word assessment than students who were not exposed to the program. However, student progress through the program was slower than initially expected by the developers. Empirical will continue to collect, integrate, and analyze outcomes for three more years.
The methodological designs for this study were presented at this year’s annual SREE conference in Washington, D.C. Results for this study will also be presented at the 2010 Annual AERA Meeting in Denver, CO. Meet the research team as they describe the study in further detail during the Division C poster session on May 3.
-
Conference Season has Arrived
March 12, 2010
Springtime marks the start of “conference season.” and Empirical Education has been busy attending and preparing for the various meetings and events. We are participating in five conferences (CoSN, SIIA, SREE, NCES-MIS, and AERA) and we hope to see some familiar faces in our travels. If you will be attending any of the following meetings, please give us a call—we’d love to schedule a time to speak with you.
CoSN
The Empirical team headed to the 2010 Consortium of School Networking conference in Washington, DC at the Omni Shoreham Hotel from February 28—March 3, 2010. We were joined by Eric Lehew, Executive Director of Learning Support Services at Poway Unified School District, who co-presented with us a poster titled, “Turning Existing Data into Research” (Monday, March 1 from 1:00pm to 2:00pm). As exhibitors, Empirical Education also hosted a 15-minute vendor demonstration entitled Building Local Capacity: Using Your Own Data Systems to Easily Measure Program Effectiveness, to launch our MeasureResults tool. Please contact us if you’d like to schedule a short webinar-style demo to take a tour of our web-based system.
SIIA
The Software & Information Industry Association held their 2010 Ed Tech Government Forum in Washington, DC on March 3–4. The focus this year was on Education Funding & Programs in a (Post) Stimulus Worldand included speakers, such as Secretary of Education, Arne Duncan and West Virginia Superintendent of Schools, Steven Paine. Check out their website for the program highlights and schedule.
SREE
Just as the SIIA Forum came to a close, the Society for Research on Educational Effectiveness held their annual conference—Research Into Practice—March 4-6 where our chief scientist, Andrew Jaciw, and research scientist, Xiaohui Zheng, presented their poster on estimating long-term program impacts when the control group joins treatment in the short-term. Dr. Jaciw was also named on a paper presentation with Rob Olsen of Abt Associates.
- Thursday March 4, 2010
3:30pm–5:00pm: Session 2
2E. Research Methodology
Examining State Assessments
Forum
Chair: Jane Hannaway, The Urban Institute
Using State Or Study-Administered Achievement Tests in Impact Evaluations
Rob Olsen and Fatih Unlu, Abt Associates and Andrew Jaciw, Empirical Education
View abstract - Friday March 5, 2010
5:00pm–7:00pm: Poster Session
Poster Session: Research Methodology
Estimating Long-Term Program Impacts When the Control Group Joins Treatment in the Short-Term: A Theoretical and Empirical Study of the Tradeoffs Between Extra- and Quasi-Experimental Estimates
Andrew Jaciw, Boya Ma, and Qingfeng Zhao, Empirical Education
View abstract
NCES-MIS
The 23rd Annual Management Information Systems (MIS) Conference was held in Phoenix, Arizona March 3-5. Co-sponsored by the Arizona Department of Education and the U.S. Department of Education’s National Center for Education Statistics (NCES), the MIS Conference brings together the people who work with information collection, management, transmittal, and reporting in school districts and state education agencies. The majority of the sessions focused on data use, data standards, statewide data systems, and data quality. For more information, refer to the program highlights.
AERA
We will have a strong showing at the American Educational Research Association’s annual conference in Denver, Colorado from Friday, April 30 through Tuesday, May 4. Please come talk to us at our poster and paper sessions. View our AERA presentation schedule to find out which of our presentations you would like to attend. And we hope to see you at our customary stylish reception Sunday evening, May 2 from 6 to 8:30—mark your calendars!
- Thursday March 4, 2010
-
MeasureResults® to be Launched at CoSN 2010
February 23, 2010
Empirical Education will launch its web-based educational research solution, MeasureResults on March 1 at the Consortium for School Networking conference in Washington, DC. MeasureResults is a suite of online tools that makes rigorous research designs and statistical processes accessible to school systems and educational publishers who want to evaluate the effectiveness of products and services aimed at improving student performance.
“MeasureResults will change the way that school districts and product developers conduct rigorous evaluations,” said Denis Newman, Empirical Education President. “Instead of hiring outside evaluators or onsite research experts or statisticians, MeasureResults allows school district personnel to design a study, collect data, and review reports in our user-friendly online platform.”
MeasureResults grew out of a federally funded research project to develop a low-cost method for schools to conduct their own research. The product was developed for commercial distribution under a Small Business Innovation Research grant from the U.S. Department of Education. By moving the educational research processes online, MeasureResults makes school-run evaluations more efficient and less expensive.
-
Poway Completes Study from MeasureResults Pilot
February 12, 2010
The results are in for Poway Unified School District’s first research study using our MeasureResults online tool. PUSD was interested in measuring the impact of CompassLearning’s Odyssey Reading program in the middle grades. Using an “interrupted time series” design with a comparison group, they found that both 7th and 8th grade students averaged 1 to 2 points higher than expected on the NWEA MAP Literacy assessment. PUSD plans to continue their evaluation of CompassLearning Odyssey in different subject areas and grade levels. Join us in D.C. at this year’s CoSN conference on March 1, 2010 as Eric Lehew, Executive Director of Learning Support Services at PUSD, presents findings and reflections on the process of using MeasureResults to conduct research at the local district level.
Click here to request a copy of the PUSD achievement report.
-
Empirical Education Appoints Chief Scientist
February 5, 2010
We are pleased to announce the appointment of Andrew Jaciw, Ph.D. as Empirical Education’s Chief Scientist. Since joining the company more than five years ago, Dr. Jaciw has guided and shaped our analytical and research design practices, infusing our experimental methodologies with the intellectual traditions of both Cronbach and Campbell. As Chief Scientist, he will continue to lead Empirical’s team of scientists setting direction for our MeasureResults evaluation and analysis processes, as well as basic research into widely applicable methodologies. Andrew received his Ph.D in Education from Stanford University.
-
Webinar: Uncovering ARRA’s Research Requirements
January 22, 2010
Researchers at Empirical Education provided a detailed overview of the various research themes and requirements of the ARRA stimulus initiatives with state department of education officials during their December 9 webinar entitled, “Meet Stimulus Funds’ Research Requirements with Confidence.“ The webinar gave specific examples of how states may start planning their applications and building research partnerships, as well as an overview of the ED’s current thinking about building local research capacity. The initiatives that were discussed included Race to the Top, Enhancing Education Through Technology, Investing in Innovation, Title I School Improvement Grants, and State Longitudinal Data Systems. A follow-up webinar was broadcasted on January 20, 2010; it outlined a specific example of a program evaluation design that districts can use with existing data. The slides and presentation of the latest broadcast can both be accessed here. Stay tuned for future webinar topics on more alternative experimental research designs.
-
Four Presentations Accepted for AERA 2010
December 1, 2009
Empirical Education will be heading to the snow-capped mountains of Denver, Colorado next April! Once again, Empirical will have a strong showing at the 2010 American Educational Research Association conference, which will be held at the Colorado Convention Center on April 30 — May 4, 2010. Our presentations will span several divisions, including Learning & Instruction; Measurement & Research Methodology; and Research, Evaluation, & Assessment in Schools. Research topics will include:
- Examining the Efficacy of a Sight-Word Reading Program for Students with Significant Cognitive Disabilities: Phase 2
- Matched Pairs, ICCs and R-squared: Lessons from Several Effectiveness Trials in Education
- Addressing Challenges of Within School Randomization
- Measuring the Impact of a Math Program As It Is Rolled Out Over Several Years
In line with our past 2 years of successful AERA receptions, Empirical Education plans to host another “meet and greet” at this year’s conference as well. Join our mailing list to receive the details.
-
Report completed on the effectiveness of MathForward
November 13, 2009
Empirical Education assisted Richardson Independent School District (RISD) in conducting an evaluation of the MathForward algebra readiness program published by Texas Instruments. RISD implemented MathForward in their 7th and 8th grade general mathematics and 9th grade Algebra I classes. The research employed an interrupted time series design comparing existing student achievement scores with MathForward to student achievement scores from the three years prior to the introduction of MathForward.
The results of this four-year study suggest that 7th grade students scored, on average, 11 percentile points higher with MathForward than the 7th grade students from the three previous years without MathForward. A similar result for the 8th grade suggests that students participating in MathForward scored, on average, 9 percentile points higher. While the trend did not hold for 9th grade, further exploration suggests that 9th grade students whose teachers had 3 years experience using MathForward scored higher than 9th grade students whose teachers did not use MathForward.
The report further illustrates how an interrupted time series design can be used to study a program as it is rolled out over several years. This research will be presented at the 2010 AERA conference in Denver, Colorado (Friday, April 30 — Tuesday, May 4).
-
Empirical Education assists the Instructional Research Group (IRG)
November 1, 2009
Starting this winter, Empirical Education will assist the Instructional Research Group (IRG) with their project, Teacher Quality Study: An Investigation of the Impacts of Teacher Study Groups as a Means to Enhance the Quality of Reading Instruction for First Graders in High Poverty Schools in Two States. The project, focusing on first-grade reading instruction in high poverty schools with large ethnic minority populations, will be funded by the U.S. Department of Education. The role Empirical Education will play in this partnership is to obtain student demographic data from IRG; to survey teachers; to enter, check, and warehouse data; and to provide data files and report summaries to IRG.
-
i3 Request for Proposals Calls for New Approaches to Rigorous Evaluation
October 26, 2009
In the strongest indication yet that the new administration is serious about learning from its multi-billion-dollar experience, the draft notice for the Invest in Innovation (i3) grants sets out new requirements for research and evaluation. While it is not surprising that the U.S. Department of Education requires scientific evidence for programs asking for funds for expansion and scaling up, it is important to note that strong evidence is now being defined not just in terms of rigorous methods but also in terms of “studies that in total include enough of the range of participants and settings to support scaling up to the State, regional, or national level.” This requirement for generalizability is a major step toward sponsoring research that has value for practical decisions. Along the same lines, high quality evaluations are those that include implementation data and performance feedback.
The draft notice also includes recognition of an important research design: “interrupted time series.” While not acceptable under the current What Works Clearinghouse criteria, this method—essentially looking for a change in a series of measures taken before and after implementing a new program—has enormous practical application for schools systems with solid longitudinal data systems.
Finally, we notice that ED is requiring that all evaluators cooperate with broader national efforts to combine evidence from multiple sources and will provide technical assistance to evaluators to assure consistency among researchers. They want to be sure at the end of the process they have useful evidence about what worked, what didn’t, and why.
-
Growth at Empirical Education is More than Just Statistical
September 1, 2009
We are growing! Empirical Education is excited to announce a recent expansion in our company. Over the summer, we have had the pleasure of adding five new members to our team, rounding out our number to 30 total employees. The addition of four members to our research department is a vital step in beginning work on our new projects taking place during the 2009-2010 school year. The statistician joining our analysis team will work closely with the engineering team to help streamline our MeasureResults processes.
In addition to enriching our staff, we are also increasing our space. We acquired a third office unit in our Palo Alto building this summer, and we are busily readying it for a move-in. Meanwhile, we are remaining quite cozy in our two current offices.
We view this expansion as a glimpse of what the future will hold for our company, and we look forward to introducing you to some of our new faces. Feel free to stop by our office anytime for a guided tour.
-
Poway District Using MeasureResults
August 17, 2009
With the new school year approaching, we are excited to announce a partnership with MeasureResults’ newest user, Poway Unified School District (PUSD). With the help of MeasureResults, PUSD educators will design and conduct their own program evaluations, while outsourcing the analytics and reporting functions to MeasureResults’ automated analysis engine. Planned study designs include an “interrupted time series,” which compares current achievement levels to levels from several years prior to the introduction of the program under evaluation. Plans also include a comparison group study which, by matching classrooms that are using the program with similar classrooms that are not, can estimate the difference that the new program has made. Special analyses will determine whether the program benefits various subgroups of students (e.g. English language learners) more than others. We anticipate that PUSD’s valuable product feedback and input will enable us to make ongoing improvements to MeasureResults’ functionality and usability.
-
Empirical Education Lends Expertise to the Software Industry
June 10, 2009
Empirical Education continued its active involvement as an associate member of Software & Information Industry Association (SIIA), participating in its 9th annual Ed Tech Industry Summit and helping to draft a set of guidelines for provider-sponsored research on educational technology products and service.
At SIIA’s San Francisco conference held May 3-5, Dr. Denis Newman discussed Empirical Education’s experience in state and local experimental evaluations as a panelist in a session addressing technology-based assessment tools entitled “Harvesting Information from Assessment to Support the Learner and the Educator.” Other panelists representing SRI and CTB/McGraw Hill brought their expertise in testing to a discussion of the differences between formative and summative assessments and how data from such assessments are used to inform decision-making at various levels: teacher, school, district, and state. Dr. Newman’s remarks drew the discussion into technology trends such as longitudinal data systems, growth scales, and value-add analysis and the need to apply statistical processes that are not yet commonly part of school data systems. The full PowerPoint presentation and summary can be found on SIIA’s webpage.
For the past year, SIIA has also supported a working group consisting of members with interest and expertise in research and evaluation. Dr. Newman was asked to participate as co-chair and has been working with the other group members to complete a set of guidelines that can provide guidance in designing scientific evaluations, given the particular characteristics of technology interventions. The company has worked with many SIIA members, assisting them to work with school systems to evaluate their products and services.
-
New Directions for Research Discussed at Institute of Education Sciences Conference
June 9, 2009
The Fourth Annual Institute of Education Sciences (IES) Research Conference was convened June 7 - 9 with an air of anticipation about new directions, as John Q. Easton began his term as director. Formerly executive director of the Consortium on Chicago School Research, he brings a new perspective to IES. While Dr. Easton is quoted in Education Week as saying he will retain the rigor that IES has made a priority, the Consortium’s work points to the importance of building local capacity for research to support reform. In a paper published online, he and his colleagues provide a clear and detailed rationale for their approach that includes the need for combining high quality research with the ability to cut through technical details to communicate both good and bad news to local decision makers.
Three Empirical Education staff members furthered this agenda of building capacity for local school and district evaluations in poster presentations at the conference. Dr. Robert Smith, the company’s vice president of engineering, outlined the company’s progress on MeasureResults™, a web-based evaluation solution for schools and districts. (Funding for the development of MeasureResults™ is from an IES Small Business Innovation Research grant.)
Dr. Denis Newman, the company’s president, and Andrew P. Jaciw, director of experimental design and analysis, presented their findings on the process of developing low cost, timely, and locally relevant experiments (funded by an IES research grant). Development efforts on MeasureResults continue through the Empirical Education team’s application of the knowledge gained from this project entitled “Low Cost Experiments to Support Local School District Decisions.” This project guides the team in developing decision makers’ understanding of, and building local capacity for, conducting evaluation research.
-
Empirical Education to Design a Multi-Year Evaluation of New GreatSchools Initiative
May 13, 2009
GreatSchools, a nonprofit provider of web-based resources for parents, has contracted with Empirical Education to design a multi-year evaluation of its initiative to empower parents to participate in their children's development and educational success.
The initiative is funded by the Gates Foundation, the Robertson Foundation, and the Walton Family Foundation. Bill Jackson, Founder and President of GreatSchools, wrote in his blog: “More than a year ago, we began to consider: What more could we at GreatSchools do to improve education? How could we do more for our large audience of parents? And what could we do for low-income parents whose children face the steepest climb to college? Our answer: We should leverage the technology of our times to create a comprehensive parent-training program and support group that inspires and guides parents — especially low-income parents — to raise children who are ready for college.”
Our evaluation will feature both formative and summative components and our planning will draw on experts from Public/Private Ventures, the Harvard Family Research Project, Stanford University, and the University of Arizona. During the evaluation’s initial year, our plan is to focus on understanding how families, community organizations, and school systems can leverage the web environment and resources being built as part of the initiative. On the basis of this theory of action, our team of researchers will operationalize the measurement of impact at the school, family, and student level, leveraging web technologies to conduct formative and summative experiments.