Curriculum Materials Matter: Evaluating the Evaluation Process
By NCTM President Diane J. Briars November 2014
Adoption of curriculum materials is one of the most important decisions a teacher, school, or district can make. While state standards describe what students are expected to learn and be able to do, what is taught in classrooms—the implemented curriculum—is heavily influenced by textbooks and other instructional materials. The instructional materials affect lesson content, depth and duration of instruction for particular topics, and topic sequence. So, while we may talk about curriculum materials as just "resources," the fact is that they strongly influence classroom instruction—for better or worse.
Not surprisingly, evaluating curriculum materials has been a hot topic of conversation at recent meetings I've attended. "Which materials are best aligned with 'the Standards'—Common Core or other state standards?" "What criteria, rubrics, or evaluation processes will result in the selection of the 'best' curriculum materials for implementing 'the Standards'?"
During my tenure as mathematics director for the Pittsburgh Public Schools, I led many mathematics materials adoption committees—and I learned a great deal about productive and nonproductive practices. From that work and my experiences with other districts and states, large-scale materials review projects, and national recommendations, I offer my "Top Lessons Learned" about effective curriculum materials evaluation.
1. Focus on the central evaluation question: What curriculum materials best support students' learning of the standards? Wording the question in terms of students' learning of content, rather than implementation of standards, puts students' learning front and center. What students learn and how well they learn it depend on both mathematics content and instruction. Framing the review in terms of students' learning makes support for effective teaching and learning a critical feature for review, along with content.
2. Remember that content analysis is much more than alignment. Alignment of content with standards is often represented through "crosswalks" that connect the two, indicating whereand when content addresses particular standards. While such an approach can be useful, effective content analysis examines how materials address standards, that is, it looks for the following:
Student Achievement Partners' Publishers Criteria provides a more detailed discussion of the preceding criteria.
3. Analyze the nature of the instructional tasks and activities—this is as important as analyzing content. This analysis examines how the materials support students' learning though opportunities to engage in tasks that promote reasoning and problem solving and teachers' implementation of effective teaching practices as described in NCTM's Principles to Actions: Ensuring Mathematical Success for All. Critical questions include the following:
Understanding the intended instructional model is essential for this analysis. Be sure to read the teacher's edition or other explanatory materials, view supporting webinars, etc., that describe the instructional model and where particular supports are located. Reviewing only the student materials may not provide sufficient understanding of how the materials are intended for use in the classroom to support an adequate analysis.
4. Focus initial reviews on student materials and teacher editions of the materials.These have the primary influence on classroom teaching and learning. Analyze ancillary materials and other supports for effective teaching and learning—such as assessments, technology integration, additional practice, and professional learning—after you have narrowed your choices to materials that adequately meet the content and instructional support criteria. All the flashy supplementary materials in the world won’t make up for flawed content or lack of high-quality instructional activities.
5. Consider equity, diversity, and access. High-quality content and instructional practices are critical for the success of all students; therefore, reviews of these aspects are essential first steps in addressing equity and access. After narrowing your choices, however, consider specific ways in which materials promote equity and access. To what extent, for example, do they—
See the CCSSO-NCSM Common Core State Standards (CCSS) Mathematics Curriculum Materials Analysis Tools for a more complete list of equity, diversity, and access criteria.
6. Recognize that all omissions or gaps are not the same. No materials are perfect. Inevitably, an evaluation process will uncover gaps, omissions, or inadequate treatment of some content. The key question is how easily teachers, the school, or the district can fill the gaps. For example, providing additional practice on a skill may be relatively easy; providing lessons to address a gap in concept development is probably more difficult. Gaps that are most difficult or impossible to fill are consistent lack of instructional tasks that engage students in problem solving, reasoning, and the mathematical practices. Expecting teachers, schools, or districts to create or find high-quality tasks for almost every lesson is unreasonable—and, most likely, will not provide the consistent quality or coherence needed for effective teaching and learning.
7. Recognize that additional content is less problematic than gaps that are difficult to fill. Given the variation in standards across states, materials are likely to contain content beyond that addressed in your standards. The issue is how that extra content affects the treatment of content addressed in the standards. If the extra content can easily be skipped, or if it contributes positively to students' learning the content addressed in the standards, then it doesn't matter. It does matter, however, when it decreases time and attention on content addressed in the standards, disrupts the focus and coherence of the materials, or is so great that the books are huge.
8. Request all series and materials produced by each publisher. When you call for materials to review, remember that some of the large companies publish more than one program, so you may have to ask to see them all. Also, request programs from smaller, alternative publishers and developers as well as the large publishers. You want to review all the options, not just the traditional best sellers.
9. Allocate sufficient time for your review process. Thoughtful analysis of the content, instructional activities, and other features of curriculum materials described above takes time. Materials that are adopted are likely to be used—and to influence instruction—for a number of years. So time spent reviewing materials carefully is time well spent.
10. Use a "narrowing choices" strategy to make the review process as efficient as possible. Clearly, thorough content analyses are time-consuming—and may seem overwhelming. To make the process manageable, first review all materials for their treatment of only one or two key content domains. Retain for further review only those materials that give adequate treatment to those domains. Then make a second cut based on your evaluation of the nature of the instructional tasks and support for effective teaching practices within those domains. After these cuts, you're likely to have a manageable number of materials for further review. For example, to review middle school materials with respect to CCSSM, you might first review all materials for their treatment of ratios and proportional relationships (grades 6 and 7) and functions and expressions and equations related to proportional relationships (grade 8). Then review materials that treat that content well from the standpoint of the nature of their instructional tasks, and so on, for that content. Submit the materials that adequately address both criteria to additional review, starting with the remaining content domains, instructional tasks, and other review criteria such as equity, diversity, and access, ancillary materials, and so on.
11. Rate and discuss rather than score. Analysis of materials is qualitative, rather than quantitative; that is, reviewers are judging the quality of content treatment, instructional activities, and so forth, in different materials. Consequently, qualitative rubrics with categories such as "Not Found," "Low," "Marginal," "Acceptable," and "High" can be more useful than numeric scales. Qualitative ratings also provide useful guidance for subsequent within- and across-grade discussions of the quality of different materials.
12. Provide adequate professional learning for the members of the review team. It is essential that all reviewers both understand the standards and are knowledgeable about the effective teaching practices for implementing them. To ensure this common base of knowledge and understanding, consider engaging reviewers in collaborative study of the standards. For CCSSM, read and analyze the progression documents in addition to the standards themselves. APrinciples to Actions book study can be a good way to build knowledge of the effective teaching practices.
13. Try out your top choices in the classroom. The real test of the quality of any materials is the learning that they support in the classroom. If at all possible, try out at least a unit or two from the materials under final consideration in several classrooms. Even if the review committee is in unanimous agreement, using the materials in some classrooms is important before finalizing the decision. When you test the materials in this way, recognize that they may use unfamiliar instructional models, so students—and teachers—will need some adjustment time. My experience has been that trying out materials has been invaluable in helping review committees adopt materials that strongly support effective teaching and learning.
A number of rubrics and tools are available to support materials evaluation. I have used the CCSSO-NCSM Common Core State Standards (CCSS) Mathematics Curriculum Materials Analysis Project Tools referred to earlier. The strengths of these tools are that they provide qualitative rubrics for analysis of different review criteria, along with worksheets that are specifically designed to support cross-grade as well as within-grade analysis of treatment of core content domains. As you consider rubrics for your process, be sure that they (1) support cross-grade analysis of content coherence as well as the quality of individual lessons or units and (2) promote discussion of strengths and weaknesses of particular materials rather than only numerical ratings.
Even though this list of review criteria and processes may seem overwhelming, in practice, these "lessons" have worked very well to guide the review process and support adoption of materials that will promote all students' learning of the standards. Selection of curriculum materials is one of the most important responsibilities of teachers, schools, and districts. And careful analysis of how materials address standards and instruction is a necessary foundation for this work and critical to the learning of all students.