Follow the Numbers
Printer Friendly VersionEmpirical Legal Studies is a relatively new trend in legal scholarship that applies scientific method to legal data. Almost two dozen faculty at the Law School have embraced this effort to test legal theory with real-world evidence. Among them, Jennifer Arlen and Geoffrey Miller are helping to spark a revolution across the legal academy.
A few years ago, Joseph Price, then a graduate student in economics at Cornell University, began building a database of basketball statistics. Price was interested in the relationship between incentives and performance, and he wanted to see whether professional players played better when their contracts were on the verge of expiring. At the time, Price also happened to be reading Blink, the best-selling book by Malcolm Gladwell, which includes a chapter arguing that most people harbor deep-seated, racist attitudes that affect their behavior. As Price read Blink, he realized that his basketball data—which included box scores from individual N.B.A. games—could be used to test Gladwell’s theory. Was it possible, Price wondered, that referees treated players differently depending on their race?
Price, now an economics professor at Brigham Young University, ended up collaborating on the research with Justin Wolfers, an economist from the Wharton School at the University of Pennsylvania. They analyzed every game over the previous 13 seasons, and they concluded that the answer to Price’s question was a clear yes. Holding all else equal—a player’s position, the location of a game and numerous other factors—the professors found that an all-white refereeing crew called between 2.5 percent and 4.5 percent more fouls per game against a black player than a white player. (Black referees, for their part, were more likely to call fouls against white players than black players, though the pattern wasn’t as strong.) “Basically,” Wolfers was quoted as saying in a front-page New York Times story last year, “it suggests that if you spray-painted one of your starters white, you’d win a few more games.”
Neither Wolfers nor Price is a lawyer, and their paper wasn’t about the law. But it did deal with the application of rules by judges, albeit basketball judges. And it addressed an issue that is central to many of today’s most contentious legal debates—namely, the extent to which race continues to play a quiet role in the administration of justice. So the paper became a main attraction at a conference that drew nearly 450 scholars to the NYU School of Law in November. They came for the second annual Conference on Empirical Legal Studies, where they reveled in law schools’ newfound interest in real-world, data-driven research. More than 100 papers were presented, on topics ranging from the impact of voter-identification laws to the pervasiveness of corporate fraud to the role that race plays in sentencing.
Empirical legal studies, often referred to as ELS, has become arguably the hottest area of legal scholarship today. Attendance at the November conference, organized by professors Jennifer Arlen and Geoffrey Miller of NYU Law, was almost twice as high as at the first conference, held at the University of Texas in 2006. A new journal—The Journal of Empirical Legal Studies—began in 2004 and now accepts less than one in 10 of the submissions it receives.
NYU, meanwhile, has become one of the centers for this new brand of empirical work. Almost two dozen members of the faculty, including Lily Batchelder in tax and social policy, Vicki Been ’83 in real estate, Marcel Kahan in corporate law, Florencia Marotta-Wurgler ’01 in commercial law, and Stephen Choi in securities law have published empirical studies in the last few years. And Arlen ’86 and Miller have played a broader role, by helping turn the recent burst of research into something of a formal movement.
In 2006, the two professors joined with Bernard Black of the University of Texas School of Law and Theodore Eisenberg and Michael Heise of Cornell Law School to build upon the foundation created by the Journal of Empirical Legal Studies two years earlier. They started the annual conference and founded the Society for Empirical Legal Studies. Arlen and Miller became the founding co-presidents of the society. “They’re very important players,” said Heise, who serves as coeditor of the Journal. “They’re engaged in their own work, and they’ve also taken on leadership roles to increase the visibility of the Empirical Legal Studies movement.”
Each has done work that has overturned preconceived notions. Miller, Stuyvesant P. Comfort Professor of Law, came to NYU from the University of Chicago in 1995 and specializes in corporate law. In 2004, he published a paper in the then-new Journal of Empirical Legal Studies, with Theodore Eisenberg, that set the conventional wisdom about class-action lawsuits on its ear. While legislators such as Senator Orrin Hatch were decrying “jackpot justice, with attorneys collecting the windfall,” the authors found the average size of class-action settlements had not, in fact, risen over the previous decade. The size of attorney’s fees in such lawsuits hadn’t risen, either. This, the professors dryly noted in their paper, “is not the sort of fact we are accustomed to hearing.”
Arlen, Norma Z. Paige Professor of Law, has taken a special interest in the sentencing guidelines governing corporate criminal liability. In t he 1990s, the U.S. Sentencing Commission adopted sentencing guidelines that constrained judges in most cases to impose higher fines on corporations convicted of crimes. In 1999, Arlen and two coauthors found that in the years after t he guidelines were adopted, corporate sanctions increased dramatically, but they also determined that the legal constraint on the judges was unnecessary. It seems that federal judges voluntarily heeded the call to increase corporate sanctions, whether or not their cases fell under the new guidelines. In 2005, the Supreme Court ruled in United States v. Booker that sentencing guidelines are no longer mandatory.
“The real importance of ELS,” Arlen said, “is that it enables us to formulate legal policy based on the real problems that exist in the world, not the problems we think might exist, based on our ideology.” The field, she added, “gets us away from anecdotes and from making policy based on which anecdote you believe.”
The empirical work on medical liability, for example, is helping to shift the debate about the role of medical error in health care. For all the talk about the soaring malpractice costs, research has shown that the main problem isn’t frivolous lawsuits; it’s widespread medical error. In 2006, for instance, Michelle Mello, an associate professor of health policy and law at the Harvard School of Public Health, testified before a U.S. House subcommittee that“only three to five percent of patients who are seriously injured by medical negligence file malpractice claims and less than half those who claim receive compensation.” In fact, other studies show that patients face a substantial risk of medical error—and support Mello’s finding that only a small fraction of those injured file lawsuits.
The 20 or so NYU Law faculty who embrace ELS are applying their data-crunching skills to other front-page issues, too. Associate Professor of Law and Public Policy Lily Batchelder, for example, testified before the Senate Finance Committee in March about the spottiness of the estate tax. Some individuals who receive extraordinarily large inheritances bear little or no tax burden, Batchelder said, while a small number who inherit relatively small amounts bear substantial tax burdens. Given that the estate tax is scheduled to disappear in 2010 but return in 2011—and that policy makers are likely to fix this oddity in some way—they have a good opportunity to make the tax fairer in the process. In effect, Batchelder is nudging the Senate to get beyond the usual ideological debate over the estate tax and to consider practical matters as well.
In May, Vicki Been, Elihu Root Professor of Law and director of the Furman Center for Real Estate and Urban Policy, testified before the House Oversight and Government Reform Committee’s domestic policy subcommittee about the effects the current wave of foreclosures are having. Been and two coauthors examined sales of properties surrounding foreclosed homes, and concluded that foreclosures significantly depress the sales prices of nearby homes. But what excited Congress and the media more is that a wholly innocent segment of the population has been adversely affected by the mortgage crisis: renters. In New York City, the Furman Center report documents, 60 percent of properties entering foreclosure in 2007 were two- to four-family or multifamily buildings, representing at least 15,000 renter households.
As Arlen notes, the empirical-research movement aims to replicate the scientific methods of the medical sciences. In those fields, researchers can investigate cause-and-effect relationships through randomized trials; some patients are given a drug, some are not, and outcomes are compared. But such trials aren’t feasible in much of the legal world. A judge can’t vary prison sentences, for instance, in order to see the effect that time behind bars has on recidivism. When legal researchers want to determine the effect of a legal change on states, they must rely on sophisticated statistical analysis to distinguish the effect of the law from other influences.
In fact, the main reason for the rise of empirical work is simply that it’s far easier to do now than it once was. Computers can crunch reams of data and allow researchers to tease out the correlations—between, say, a defendant’s skin color and his sentence length—that once would have remained hidden. “You can do work on your laptop today,” Miller says, “that would only have been possible on a mainframe 15 years ago.”
But the empirical movement has also come along at a serendipitous time in the intellectual cycle. The legal fields that were growing in the 1980s and 1990s don’t have quite the energy that they once did. These fields included law and economics (which mostly attracted professors on the right side of the political spectrum) and critical legal studies (which attracted those on the left). By the current decade, the arguments of those fields no longer seemed so new, and young professors discovered that they could more easily make their mark not by offering new theories to explain the world but by investigating what was actually occurring.
Miller notes that the most coveted faculty recruits once had to just be fantastically smart lawyers, like Supreme Court clerks; today, the schools want not only brain power, published papers and impressive credentials, but also research experience in the social sciences. “There have always been people who looked at data, at least since the 1930s,” he said. “But in the last 10 years, it’s become probably the most important development in legal studies.”
ELS has made ripples in Washington, as Batchelder’s and Been’s appearances before Congress suggest. And some of its findings—like those on the prevalence of medical error—have helped support efforts to change policy. But the field’s overall effect on policy—a clear goal of ELS proponents—has been tricky to measure. Part of that is merely a reflection of the field’s youth. But part of it, some scholars say, stems from the fact that doing truly unassailable empirical research is so difficult. “The question is, ‘How good is this stuff?’” said noted legal theorist and law-and-economics proponent Richard Epstein, a visiting professor at NYU who attended the November ELS conference, but has not done empirical work himself. “I have mixed emotions.”
One problem is finding enough relevant data. As Arlen says, “We have too little data to examine many of the issues we care about.” Another is designing a study that enables researchers to isolate the effect of a change in the law from all other potential causes of change. As a result, it is not uncommon to get multiple studies of the same topic with differing results. The best example may be the recent dueling studies over the effect of the death penalty, which have been covered in the mass media. Some studies have confidently declared that the death penalty causes a reduction in murders in the states that impose it. Other papers, just as confidently, say that the amount of noise in the data makes it impossible to conclude that the death penalty is a deterrent.
Yet there is also a broad swath of work that gets nearly universal praise even from skeptics like Epstein. In the end, then, the way forward certainly involves more empirical work, so that the compelling research can ultimately win out over the flawed studies—and so that legal scholars, lawyers, judges and policy makers can get a better understanding of how the law actually affects people in their day-to-day lives.
“Theory is just theory,” as Miller says, “but data is something policy makers take seriously.”
—
