Concerns Regarding the Use of EdReports Mathematics Materials Reviews
An Open Letter to the Education Community
The Common Core State Standards for Mathematics (CCSSM) and other college- and career-ready standards provide a strong foundation for improving mathematics teaching and learning across the United States. One of the most important requirements for realizing the potential offered by the standards is the selection of instructional materials that support effective implementation.
Recently a new organization, EdReports, has undertaken the work of reviewing instructional materials for their “alignment to the Common Core and other indicators of high quality . . . so that teachers, principals, district and state officials charged with purchasing materials can make more informed choices” when doing so. Over the last year, EdReports has created an instructional materials evaluation tool, process, and reporting format that are intended to identify materials aligned with the CCSSM. Reviews of 20 elementary and middle school mathematics series are now posted on the EdReports website, along with details of the review methodology. Summaries of publishers' responses to these reviews are also posted.
Unfortunately, the EdReports methodology, including its evaluation tool and process, has produced reviews that fall short of providing useful and accurate information about many critical features of materials reviewed, such as how the materials address the Standards for Mathematical Practice and the quality of the instructional activities. As a result, the current ratings and reviews do not provide the types and quality of information needed to make informed choices about the extent to which particular materials support students' learning, or teachers' teaching, of CCSSM.
The following features of the current EdReports methodology are of particular concern and have produced results that are at odds with its stated mission of identifying high-quality instructional materials that support implementation of college and career standards.
1. The EdReports alignment criteria focus on only a subset of the CCSSM standards for each grade, which they term Major Work, and require that materials “devote the large majority of class time” (65-85 percent) to this subset of standards. This departure from CCSSM is problematic because Major Work not only excludes some CCSSM standards, but also excludes standards in Critical Areas specified in CCSSM-for example, geometry in K-grade 5 and probability and statistics in grades 6-8. (The accompanying table is a side-by-side comparison of CCSSM Critical Areas and EdReports Major Work.) Materials that address the full set of standards as specified in CCSSM are deemed out of alignment because they devote too many lessons to key CCSSM content that EdReports does not consider Major Work. Moreover, materials that EdReports identifies as meeting its criteria are likely not to include sufficient attention to key CCSSM Critical Areas for each grade. A thoughtful analysis of materials to support implementation of CCSSM needs to examine the full set of CCSSM standards, with special attention to those in all the Critical Areas. Furthermore, the analysis needs to focus on the quality of the treatment of the standards instead of on whether the number of lessons devoted to a subset of standards satisfies an arbitrary criterion.
2. The EdReports Evaluation Tool specifies three major gateways that materials must clear sequentially. Review of instructional material stops and does not continue to the remaining gateways as soon as it fails to meet the criteria of a single gateway. The result is that only materials that meet the criteria for Gateway 1 (26 out of 87 grade-level materials) are reviewed regarding treatment of the Standards for Mathematical Practice and other critical features specified for Gateway 2. Then, only materials that meet the criteria for Gateways 1 and 2 (11 of the 87 grade-level materials) are reviewed for quality of instructional activities. Because of this gateway system, only 11 of the 87 grade-level EdReports reviews include analysis of all the relevant features of instructional materials. This is a critical omission for educators attempting to make wise decisions about the quality of instructional materials. Every review should be complete.
3. The EdReports process currently allows reports to be posted with numerous errors. Some of these reflect a misunderstanding of CCSSM, others are mathematical errors, and some are related to an understanding of effective teaching and assessment.1 Any useful analysis should be free of errors.
Determining the extent to which materials support student learning of a set of college- and career-ready standards is time-consuming and demanding work. Research and experience have taught us that identifying instructional materials with the potential to support learning for a broad range of students requires an analysis of the content (skills, concepts and practices), the nature of tasks, lessons, and assessments, and finally the sequencing of content within and across grades/courses. Individually and together, these features contribute to or undermine the opportunity for students to learn.
Any effective review methodology will incorporate each of these elements. However, the EdReports alignment criteria redefine critical CCSSM content for grade-level focus, the evaluation tool prevents an analysis of the Standards for Mathematical Practice for some materials, and the current process fails to ensure that reviews are free from mathematical errors. The result is a set of incomplete reports with inaccurate conclusions and misleading recommendations.
The National Council of Teachers of Mathematics and the National Council of Supervisors of Mathematics wholeheartedly support the goals of EdReports. However, its reviews are leading educators who are analyzing how instructional materials support implementation of CCSSM to make decisions that will do a disservice to our students and will squander the potential of the standards to improve mathematics education.
Finally, we strongly encourage EdReports to revise its current methodology, evaluation tool, and process to address the issues raised above and to withdraw the current reviews while these improvements are being made. Materials then should be reviewed again under the new criteria so that comprehensive and error-free reviews are available before any work is begun on additional materials. Making these changes will increase the likelihood that EdReports reviews will be valuable for educators attempting to make informed choices about the purchase or use of instructional materials to support student learning.
We, and our organizations, support this letter:
Association of Mathematics Teacher Educators
Christine D. Thomas, President
Benjamin Banneker Association
Margaret J. Walker, President
TODOS: Mathematics for ALL
Susie W. Håkansson, President
1 For example, the EdReports Connected Mathematics Project Grade 7 review states, “Lines in Grade 7 should go through the origin” and penalizes these materials for having “many items with the y-intercept that are [sic] not at 0” (p. 2). The relevant CCSSM standard is 7.RP.A.2a: “Decide whether two quantities are in a proportional relationship, e.g., by … graphing on a coordinate plane and observing whether the graph is a straight line through the origin.” which clearly requires working with both lines that go through the origin and those that do not.
View chart: CCSSM Standards in Critical Areas Compared with those in Major Work as Specified in EdReports Evaluation
View a PDF of this statement and read the news release.