On the Wisdom of Crowds

Social software developers should remember the many caveats of Surowiecki's theory.

In "The Wisdom of Crowds," James Surowiecki argues that, under certain conditions, the collective judgements of large groups of people are more accurate than the judgements of any individual, even an expert. [1] The idea holds so much appeal for the social software community that "the wisdom of crowds" has become a part of the vernacular. It is invoked as a source of added value in news aggregators, folksonomies, and wikis.

Unfortunately, most of these claims are misguided. The problem is that people tend to forget the "under certain conditions" part of Surowiecki's theory. The effects he describes only occur when members of the crowd in question:

  1. Don't communicate with each other,
  2. Hold diverse opinions, and
  3. Are presented with a clearly-defined problem.

The third and first conditions run counter to core goals for most social websites. The problem to be solved is usually a question like "What's the best content" or "Which movies will I like?" Subjective judgments like these cannot be clearly defined, so websites built around them are not compatible with the wisdom of crowds.

More importantly, social software is all about communication. Exposing other users' ratings data (or any other kind of feedback) biases subsequent judgments. A site built to exploit the wisdom of crowds would not allow users to see peer ratings, or any other kind of feedback, before they had reported their own opinions. That would make review and aggregator sites pretty useless, which may be why no one does it that way.

These two issues make Surowiecki's thesis irrelevant to the bulk of social software applications. However, there is a far more fundamental problem with the theory, and it's rooted in the second condition: diversity of opinion. In short, he grossly over-extends his argument when he claims that crowds are more "intelligent" than individuals.

The Law of Large Numbers

Surowiecki's archetypal example comes from a 1906 county fair where 800 people participated in a contest to guess what the weight of an ox would be after it was butchered. The average guess was 1,197 pounds. The actual weight turned out to be 1,198 pounds. On its face, this seems like a dramatic testament to the ideals of democracy, but the accuracy of the average guess has much more to do with the nature of the problem than with the wisdom of the crowd.

Their task was clearly-defined and required no special information. Each person was free to guess any weight they wanted, but the higher or lower their guess, the more obviously wrong it would be. Random variation ensured that every high guess was counter-balanced by a low guess that was equally off the mark. After 800 such guesses, the average would stick right in the middle. In this case, the average happened to be the truth.

You can tease the same kind of wisdom out of a handful of dice. Say you hold a contest to guess the number you're thinking of: 3.5. Only six-sided dice can enter this contest and, therefore, all guesses will range from 1-6. (Note that each die is physically incapable of guessing correctly, as dice can only express whole numbers.) Each die can enter the contest as many times as it wants and, eventually, you gather several hundred entries. Miraculously, the average "guess" is exactly 3.5! Again, the average just happens to be the truth.

The trick is that truly diverse (i.e. random) opinions will always vary around the mean. When you aggregate a whole lot of random opinions, you get a deceptively precise average, but this is not "wisdom" in any real sense. It's a statistical artifact called the Law of Large Numbers and it has nothing to do with intelligence.

There are two frustratingly common factors that throw this trick right off the rails. The first is communication, as discussed above. It leads to the primacy effects and power law distributions that plague news aggregator sites. The second is bias that arises from common wisdom... or lack thereof.

What if you asked a crowd to answer the following well-defined question: "What is the distance to Alpha Centauri?" Because astronomical distances are so much larger than anything in a normal person's experience, their guesses would probably fall short of 25 trillion miles. (An astronomer, on the other hand, would be right on the money.) In this case, the average just isn't the truth.

The Folly of Crowds

Crowds only appear wise when individual judgments vary randomly around the truth. Introduce any kind of systematic bias and the whole thing comes crashing down. Surowiecki cites many potential sources of bias in his book, and social software developers would be wise to take note. Social psychology offers many more: group polarization, conformity, in-group bias, framing, groupthink, and on and on. The real science is all about why groups are dumb.

References

  1. Surowiecki, J. (2004). The Wisdom of Crowds. New York: DoubleDay.