The College Fix

UPDATED to include a response by Professor Scott Page

A widely touted study claiming diversity is a better attribute than ability in spurring productivity and problem solving – claims used to push diversity classes as graduation mandates at universities – is actually fundamentally flawed, two mathematicians tell The College Fix.

University of Michigan Professor Scott Page, director of the campus’ Center for the Study of Complex Systems, authored the book “The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools and Societies” in 2007 which states “the veracity of the diversity trumps ability claim is not a matter of dispute. It’s true, just as 1+1=2 is true.”

Page’s work has since been cited by universities such as UCLA to create new diversity graduation requirements, and has also been touted by the likes of the U.S. Geological Survey, the Lawrence Berkeley Labs, and even NASA.

It’s a model that has been used “to give a scientific veneer to the diversity field,” according to Professor  Abigail Thompson, a mathematician at UC Davis.

The problem is, she adds, the model is “an example of the misuse of mathematics in the social sciences.” Thompson makes that point in her peer-reviewed critique of the model published September by the American Mathematical Society.

“The paper ‘Groups of diverse problem solvers can outperform groups of high ability problem solvers’ contains a theorem that has neither mathematical content nor real-world applications, and a contrived computer simulation that illustrates the well-known fact that random algorithms are often effective,” Thompson stated in her extensive rebuttal. “What the paper emphatically does not contain is information that can be applied to any real-world situation involving actual people.”

Colgate University’s math department chair Professor Aaron Robertson, in an email to The College Fix, said that as a mathematician he corroborates Thompson’s work. Page and the co-author of his model, Loyola University Chicago economist Lu Hong, are two non-mathematicians who attempted to prove that diversity trumps ability “through a mathematical framework mired in unnecessary mathematical jargon,” Robertson said.

“Hong and Page’s ‘study’ is routinely pointed to as scientific proof that diversity – in and of itself – is a sign of excellence. Because … science,” Robertson said. “We must make it known that Hong and Page’s ‘results’ are actually vacuous (or, at best, highly trivial) statements.”

Bad social science based on faulty math

“Diverse groups of people bring to organizations more and different ways of seeing a problem and, thus, faster/better ways of solving it,” Page told The New York Times in 2008.

“The reason: the diverse groups got stuck less often than the smart individuals, who tended to think similarly,” Page said. “What the model showed was that diverse groups of problem solvers outperformed the groups of the best individuals at solving problems.”

But Thompson’s paper in the Notices of the American Mathematical Society systematically dismantles Page’s sophistic mathematics.

First, she shows that the theorem Page and Hong establish that diversity is important in the functioning of a group of people is not only mathematically incorrect, but also trivial and irrelevant to the computational experiment the authors run to test their hypothesis.

Second, she shows that the computational experiment they employ is “contrived,” replete with artificial “restrictions on the algorithms” that essentially make sure the experiment returns to the authors the results they wish to see, and an “attempt to equate mathematical quantities with human attributes,” which is “inappropriate.”

Not only is the purely theoretical study flawed in its theory and in its computer simulations, but it is entirely devoid of any empirical evidence that deals with flesh-and-blood human beings.

Professor Robertson adds that Thompson’s paper shows how Page assumes one person from the group cannot finish the task alone, and that the entire group will always finish the task, therefore the entire group will outperform the individual.

“There are many quantifiers and the use of probability,” he said. “But … Page cannot even apply the theorem correctly because of a lack of understanding of the existential quantifiers.”

Page essentially states in his book that it is better to choose job applicants at random than attempt to “rank people by some crude ability score and pick the best” – try to float that idea at your next hiring meeting, Robertson said.

“It is unfortunate that Hong and Page’s article was published in the (what I used to think was prestigious) Proceedings of the National Academy of Sciences,” he adds. “One is left to ponder: Did any mathematician referee this article, and, how much does diversity trump truth?”

Studies contrived by social sciences

Page essentially toiled away in the ivory tower contriving a tortured mathematical simulation that used letters and numbers as proxies for human beings, did a methodologically flawed computer experiment, and then claimed, without a shadow of a doubt, that his theorem is a “mathematical truth” which has real-world applications.

And people, being as they are unaccustomed to reading and critically analyzing ethereal mathematical papers, tend to take such findings as truth merely because the author can write those three little letters, PhD, after his name.

This is a huge problem. For example, UCLA’s approval of a new diversity requirement was entirely based on social science studies that used bad science, emotion and rhetoric.

The UCLA committee that drafted the diversity proposal bombards the reader with a slew of studies that use inaccessible jargon and claim plausible conclusions, which, upon intense scrutiny of the methodologies used, tend to fall apart.

The problem is most people don’t have the time for such intense scrutiny and thus end up deferring to the PhD’s who they assume are infallible experts in their particular fields. And in turn, the social sciences are used to push a political and philosophical agenda on the masses.

College Fix contributor Josh Hedtke is a student at UCLA.

Professor Page responds:

Ability Can Trump Diversity (It also may not)
A Comment on the Role of Formal Models in Social Science

A recent article in Education News discussed my research on the value of cognitive diversity in collective problem solving. The article highlighted a result that I had derived with Lu Hong that characterizes a set of sufficient conditions for a randomly selected group of problem solvers to always outperform a group consisting of the best problem solvers. The conditions, as one might expect, are rather strong given the random group must always do better.

The purpose of proving extreme results such as this are two fold. First, to see if it’s possible – could a random group really be better than a group of the best? And second, outcomes are often continuous — meaning that if there is a set where the random group always does better, that set is most likely surrounded by a much larger set where the random group usually does better. In the case of the model in question, that conjecture appears to be true, as Lu and I show by presenting simulation results in a region not covered by the theorem.

As also noted in the Education News article, a recent a paper by a mathematician pointed out that in our sketch of the more general proof, we had neglected to include our assumption that no two solutions have the exact same value. This is a zero probability event assuming random functions. By convention social scientists exclude it, so as not to spend our time worrying about angels dancing on the heads of pins. Without that explicit assumption, a clever person can maneuver the dancing angels so that the result doesn’t hold.

These technical details aside, the mathematical fact remains that one can construct any number of sufficient conditions for diversity to either trump or defeat ability. The union of these sets constitutes a rather large portion of the space of possible instances of the model. That’s why we wrote the paper. We were surprised at how often the diverse groups won.

But, there’s a much larger point here that merits the attention of the readers of Education Week that relates to the purposes of formal models. Yes, we can subject them for empirical testing, but we can also use them to change how we think about the world. The Hong Page, Perspective–Heuristic model forces us to think about people as collections of tools and of ability as a reflection of the applicability of those tools to a given set of problems. This is an idea that goes back to Hayek, but one that had lacked a formalization. (For more details see my book The Difference.)

Within the Hong Page framework, a group’s collective potential to solve difficult problems depends on both ability (people with lots of tools) as well as diversity (people with distinct tools). Strong empirical evidence supports that hypothesis. Further, the Hong Page framework expands on the notion of ability as some sort of general capacity and in doing so provides a language with which intelligent, thoughtful people can discuss a variety of issues related to hiring, inclusivity, and opportunity. That, I believe, is worthy of Education News.

IMAGE: AJCI /Flickr

Like The College Fix on Facebook / Follow us on Twitter