A Policy Backgrounder from
The Association of Educational Publishers
January 2, 2003

No Child Left Behind (NCLB): Most Frequently Asked Questions

Introduction
An appropriations bill containing the largest-ever dollar increase in federal education spending accompanied the current iteration of the Elementary and Secondary Education Act, known as No Child Left Behind (NCLB). NCLB's influences on educational publishing include its best-known tenets—required yearly testing for grades 3 to 8, "highly qualified" teachers in every classroom—as well as its philosophy: the law's insistence, mentioned more than 100 times, on "scientifically based research" as the key to making sure education dollars are well-spent. Below, we review NCLB's biggest effects on the industry, noting also those that remain uncertain.

What are the biggest changes?
Title I funding greatly increased and, with its new "targeted assistance" for high-poverty schools, rewarded large, urban districts most. Of particular interest is the $900 million K-3 program Reading First. Reading First is a formula program for all states: As their applications are approved, states will receive federal funds (based on their number of children in poverty) and will distribute money to eligible local agencies competitively. (See links on our resource list, for more details on Reading First.)

For "low-performing" Title I schools—those that have been at a failing status for at least three years—the district must use at least 5 (and up to 15) percent of the Title I allocation to enlist the services of an outside service provider or to bring in a supplemental curriculum. The service provider must be approved by the state, and some have assembled lists. (See below.) Struggling schools are required to bring in "technical assistance" to help them analyze achievement data, identifying and addressing areas of need, and to help them pick and implement professional development and instructional strategies.

In the after-school program 21st Century Schools, funded at $1 billion, funds now flow through the state agencies, rather than directly to districts from the Education Department. State agencies are accountable for results and, as in Reading First, may be more prescriptive.

NCLB's testing requirements, of course, represent one of the industry's biggest changes. Though the requirements don't go into effect until 2005, last year $387 million in federal money was appropriated to help states come into compliance. If a state already is in compliance, however, it can take its $3 million and use it for other purposes—including implementation of an online assessment system.

What is “scientifically-based research”?
As states and districts are called upon by the federal government to show a "scientific basis" for what they do in education, it falls to publishers, in turn, to show the states and districts their products and approaches are supported by SBR [scientifically based research]. Just what is SBR?

All federal sources agree that the "gold standard" for scientific research, in education and elsewhere, is the randomized, controlled trial that systematically tests a clearly defined "causal claim" for the approach, program, or materials in question, showing valid, reliable data and analyses across evaluators and observers and across multiple measurements. Independent, or third-party, review is expected.

Even the NCLB definition, however, does leave the door open to quasi-experimental design. For instance, LeapFrog SchoolHouse, whose design was approved by the National Institute for Child Health and Human Development, randomly chose which schools would get its Literacy Center and which would be "controls"—rather than randomly assigning individual students or classes. Reliability does seem to mean big numbers over time: LF is carrying out two, two-year studies, each involving a total of 200-400 children.

Is there more than one model that qualifies as SBR?
In her paper for USED's 2/02 seminar on SBR, Valerie Reyna of OERI (now the Institute of Education Sciences) set up this hierarchy for various "levels of evidence":
-randomized trial
-quasi-experiment, including before and after
-correlational study with statistical controls
-correlational study without statistical controls (such as controls for class size)
-case studies

Experts agree that in many areas of education, there is no "gold standard" research available. Many think that the only established examples are those of the National Reading Panel on early reading and of the long-term study of reduced class size in Tennessee. In other areas, publishers, like education agencies, should present the best evidence they have, while making plans to move their evidence up the hierarchy. There is no deadline by which everyone is required to have reached the gold standard, but those who can get closest to the ideal most quickly will have an advantage. AEP is asking ED to provide guidance to supplemental publishers to help them make strategic choices about research.

Does the SBR requirement apply to supplemental materials?
The direct response AEP has gotten so far relates to Reading First, where, USED's Susan Neuman reiterated, the SBR requirement applies to all materials and approaches. We believe we can extrapolate that, for any component whose NCLB language uses this phrase, supplementals will need to demonstrate SBR at the best level they can. However, the evidence you're required to present will be in proportion to the claim you make; assuming your claim is more circumscribed than that of a core-curriculum publisher, so can be your research.Does SBR apply to all grades and subject areas?

Again, there is a more detailed knowledge base in some areas—such as early reading—than in others. We believe that where the knowledge base does not yet exist, standards for acceptable research will be more flexible, initially. But given the Fed's traditional "bully pulpit" role, publishers should recognize SBR as the trend for all areas and grades.


Can schools buy from publishers in the meantime as they gather evidence?
No one is prohibited from buying, but the burden is on schools and districts to back up the choices they make with the best evidence available. Again, provide the best evidence you have.

To see what kinds of evidence are considered sufficient for education agencies to show, we recommend looking at an approved Reading First grant, in a state that you are targeting, or in one whose profile is similar to one you are targeting. (See list.)

What about materials teachers pay for with their own money?
Any purchases not made with federal dollars are not governed by NCLB's ideas—but may be influenced by them, nonetheless.

Is secondary research acceptable?
An important, and still-open, question. A related question is whether the phrase "research based" has a different, and less stringent, interpretation. As part of our CEO Roundtable panel in June 2002, John Bailey of the Education Department commented that the software industry was able to get criteria for research on the Enhancing Education Through Technology program changed from "scientifically based" to "research based," so that software publishers could cite their use of practices proven by third parties— without conducting rigorous primary research on their products. Whether the two phrases continue to have distinct meanings, and how "research based" might apply to supplementals, remains in question.

Is there or will there be a federal list of approved materials?
The federal government has stated it does not intend to make a list of "allowed" materials, even in Reading First. The closest thing, on the federal level, is the resource listing known as the What Works Clearinghouse. The WWC recently has closed the period of public comment on its criteria for research, known as the Design and Implementation Assessment Device (DIAD). The DIAD is expected to be finalized this month (January 2003) with review of individual studies slated to begin in the spring.

The DIAD's stated purpose is to "provide a standardized method for evaluating the validity of causal claims in individual studies"; three phases of questioning, each at a different level of specificity, are involved in producing an "evidence report" for each piece of research reviewed. For more details on the DIAD, and for a flow chart of WWC activities and dates, consult the WWC Web site (link is included on our resource list).


Are states forming lists?
In some areas—such as supplemental service providers, required for "low-performing" schools—states, as we've mentioned, must approve providers, and some have made lists. With some, the process is to go to vendors for bids and then select based on state criteria. Some rely most on local agencies—for advice on what the LEAs have found most successful, or as service providers themselves. (LEAs can offer bids.) Providers that show a track record, easy implementation, and plenty of support will do best, say state officials.


Will ED provide funding for research?
A recent report by the Coalition for Evidence Based Policy—a nonprofit organization led by a bipartisan board—has called for the Department to make a major, sustained commitment to fund education studies that use random assignment and that make comparisons between control groups and student groups "treated" with particular educational interventions. The report was produced with help from ED; its effects remain to be seen. (See resource list for link.)

Will ED help with research design?
Resources we know of so far include the National Institute for Child Health and Human Development. At a recent meeting of the Partnership for Reading (AEP has joined as a "Working Partner," advising on the direction of this collaboration among NICHD, USED, and the National Institute for Literacy), Reid Lyon of NICHD offered that the agency would consult with publishers interested in designing original research.

Asked to discuss how the Partnership plans to work with educational publishers to ensure that instructional materials meet scientifically based requirements, NIFL Director Sandra Baxter has responded that periodic meetings of the Working Partners Group are designed to help the Partnership "think through how best to meet this challenge."

Further Resources