In a recent issue of Time, Joe Klein acknowledges the ignored reality that national-scale programs based on effective pilot programs frequently do not yield the same successful results. His case in point is Head Start—a “Great Society” pre-school program intended to provide a boost to disadvantaged children before they enter elementary school.

Head Start was based on a few pilot programs, such as the Perry Preschool program, that were believed to be effective. Advocates asserted that a national preschool program for disadvantaged children would yield the same positive results.

However, the 2010 Head Start Impact Study, a scientifically rigorous evaluation of multiple Head Start sites throughout the nation, found that the program is clearly ineffective. The program has had little to no positive effects for children granted access to Head Start.

Klein asks, “Why do so many [government programs] succeed as pilots and fail when taken to scale?” The answer to his question is two-fold. First, national programs often do a poor job of replicating the crucial factors found in the pilot programs that are necessary for producing the same successful results, such as hiring highly skilled staff. Second, the social conditions contributing to the success of a particular pilot program are often not present in other settings. A very poignant example is the case of police departments performing mandatory arrests in domestic violence incidents.

During the 1980s, criminologists Lawrence W. Sherman and Richard A. Berk (currently professors at the University of Pennsylvania) analyzed the impact of mandatory arrests for domestic violence incidents on future domestic violence incidents in Minneapolis, Minnesota. The experiment found that mandatory arrests led to significantly lower rates of domestic violence. Police departments from across the nation adopted the mandatory arrest policy based on the results of this lone program conducted in a single city. Did the positive results hold when mandatory arrest policies were replicated and evaluated in other cities?

What worked in Minneapolis did not always work in other locations. Replications in Omaha, Nebraska; Milwaukee, Wisconsin; and Charlotte, North Carolina; found that mandatory arrests lead to long-term increases in domestic violence. Apparently, re-offenders, knowing that they would be automatically arrested and spend the night in jail, responded by becoming even more abusive to their partners.

A subsequent analysis by Sherman in his book Domestic Violence: Experiments and Dilemmas postulated that arrested individuals lacking a stake in conformity within their communities were significantly more likely to engage in domestic violence after arrest, while married and employed arrested individuals were significantly less likely to commit further domestic violence infractions. Thus, the social conditions in Omaha, Milwaukee, and Charlotte led to an entirely different result than in Minneapolis.

Policymakers and advocates of social programs often assume that a single social program found to be effective in a single setting will automatically have the same results when implemented in other settings. This assumption too frequently turns out to be dead wrong.