More Random than Random

And here you are with the final sample article I wrote up:

Quick quiz: What do streak-shooting basketball players, Princeton’s Global Consciousness Project, and iTunes’ Smart Shuffle feature have in common. If you said, “They’re all the result of people misinterpreting random data,” then you’re reading this out loud and probably getting strange looks right about now (You’re also right).

The human mind doesn’t do well with the random, but, as with most flaws in human perception, there’s actually a good reason for this. In harsher times, when survival was less of a given than it is today, there was a distinct evolutionary advantage to finding patterns. A human who noticed many patterns was more likely to spot the pattern indicative of a predator, or realize that a seed dropped into the ground last year had turned into a sprout this year. A few extra patterns spotted may incur slight inconveniences, but it’s still a lot better than dying because you missed one.

The flip side to this is that now humans aren’t good judges of random data, and start to see patterns where there are none. For instance, take the following three strings of numbers, representing a random choice from 4 numbers:

12414412
23123441
21434231

One of them is completely random, the other two aren’t. Which one do you think is the random one? Human instincts lead most people towards the second or third choice, but it’s actually the first one—which deviates the most from the expected distribution and looks streakiest—which I generated randomly. It has too many 4’s and 1’s and absolutely no 3’s, but that was the way the dice rolled. In the long run, random data will indeed average out, but streaks are to be expected in the short term.

So let’s go back to the examples I listed at the beginning of this article. It’s a commonly-held belief that in basketball, players are prone to streak shooting, often referred to as the “hot hand.” After a few successful shots, a player is believed to “loosen up” or “get into a groove.” The reverse supposedly happen after a few misses. To determine whether or not this actually happens, Gilovich et al. examined the records of the Philadelphia 76ers during the ’80-’81 season. The results were surprising, to say the least: It turned out that after a series of baskets, players were less likely to score again. On the other hand, after a series of misses, players were more likely to land a basket. What was really going on is that people were noticing the streaks that inevitably arise in random data and believing them to be non-random.

A similar situation arose with the shuffle feature on iPods at one point. Apple received numerous complaints and inquiries from people who claimed that their iPods were playing favorites. Some artists or albums seemed to show up much too often, while others too rarely—if at all. Their software engineers went over the random number generator again and again, and they found nothing wrong with it. And yet people wouldn’t believe them. Eventually, they decided they had to address it somehow, so they came out with the “smart shuffle” feature. It allowed people to decrease the likelihood that multiple songs by the same artist would be heard within a small amount of time. As Steve Jobs, Apple co-founder described it, “We're making it less random to make it feel more random.”

My final example is the worst of the bunch: Princeton’s Global Consciousness Project. This project was inspired by work done in PEAR (The Princeton Engineering Anomalies Research), and it essentially boils down to people watching random numbers and trying to judge if they see deviations during major world events. And, surprising as it may not be, they’re able to find deviations from a perfectly chance distribution around every major event (and plenty of deviations around absolutely nothing, but they don’t talk about them). They claim that these deviations are due to some “quantum energy field” in the world caused by human consciousness. In the eight years that the project has been active, all it's accomplished is spotting streaks in random data while wasting mass amounts of time and money, all of which could have been saved with just a little bit better understanding of what 'random' means, but no one's stepped in to put a stop to it, so far.