“EARTH has its boundaries, but human stupidity is limitless,” wrote Gustave Flaubert. He was almost unhinged by the fact. Colourful fulminations about his fatuous peers filled his many letters to Louise Colet, the French poet who inspired his novel Madame Bovary. He saw stupidity everywhere, from the gossip of middle-class busybodies to the lectures of academics. Not even Voltaire escaped his critical eye. Consumed by this obsession, he devoted his final years to collecting thousands of examples for a kind of encyclopedia of stupidity. He died before his magnum opus was complete, and some attribute his sudden death, aged 58, to the frustration of researching the book.

Documenting the extent of human stupidity may itself seem a fool’s errand, which could explain why studies of human intellect have tended to focus on the high end of the intelligence spectrum. And yet, the sheer breadth of that spectrum raises many intriguing questions. If being smart is such an overwhelming advantage, for instance, why aren’t we all uniformly intelligent? Or are there drawbacks to being clever that sometimes give slower thinkers the upper hand? And why are even the smartest people prone to – well, stupidity?

It turns out that our usual measures of intelligence – particularly IQ – have very little to do with the kind of irrational, illogical behaviours that so enraged Flaubert. You really can be highly intelligent, and at the same time very stupid. Understanding the factors that lead clever people to make bad decisions is beginning to shed light on many of society’s biggest catastrophes, including the recent economic crisis. More intriguingly, the latest research may suggest ways to evade a condition that can plague us all.

The idea that intelligence and stupidity are simply opposing ends of a single spectrum is a surprisingly modern one. The Renaissance theologian Erasmus painted Folly – or Stultitia in Latin – as a distinct entity in her own right, descended from the god of wealth and the nymph of youth; others saw it as a combination of vanity, stubbornness and imitation. It was only in the middle of the 18th century that stupidity became conflated with mediocre intelligence, says Matthijs van Boxsel, a Dutch historian who has written many books about stupidity. “Around that time, the bourgeoisie rose to power, and reason became a new norm with the Enlightenment,” he says. “That put every man in charge of his own fate.”

Modern attempts to study variations in human ability tended to focus on IQ tests that put a single number on someone’s mental capacity. They are perhaps best recognised as a measure of abstract reasoning, says psychologist Richard Nisbett at the University of Michigan in Ann Arbor. “If you have an IQ of 120, calculus is easy. If it’s 100, you can learn it but you’ll have to be motivated to put in a lot of work. If your IQ is 70, you have no chance of grasping calculus.” The measure seems to predict academic and professional success.

Various factors will determine where you lie on the IQ scale. Possibly a third of the variation in our intelligence is down to the environment in which we grow up – nutrition and education, for example. Genes, meanwhile, contribute more than 40 per cent of the differences between two people.

These differences may manifest themselves in our brain’s wiring. Smarter brains seem to have more efficient networks of connections between neurons. That may determine how well someone is able to use their short-term “working” memory to link disparate ideas and quickly access problem-solving strategies, says Jennie Ferrell, a psychologist at the University of the West of England in Bristol. “Those neural connections are the biological basis for making efficient mental connections.”

This variation in intelligence has led some to wonder whether superior brain power comes at a cost – otherwise, why haven’t we all evolved to be geniuses? Unfortunately, evidence is in short supply. For instance, some proposed that depression may be more common among more intelligent people, leading to higher suicide rates, but no studies have managed to support the idea. One of the only studies to report a downside to intelligence found that soldiers with higher IQs were more likely to die during the second world war. The effect was slight, however, and other factors might have skewed the data.

Intellectual wasteland

Alternatively, the variation in our intelligence may have arisen from a process called “genetic drift”, after human civilisation eased the challenges driving the evolution of our brains. Gerald Crabtree at Stanford University in California is one of the leading proponents of this idea. He points out that our intelligence depends on around 2000 to 5000 constantly mutating genes. In the distant past, people whose mutations had slowed their intellect would not have survived to pass on their genes; but Crabtree suggests that as human societies became more collaborative, slower thinkers were able to piggyback on the success of those with higher intellect. In fact, he says, someone plucked from 1000 BC and placed in modern society, would be “among the brightest and most intellectually alive of our colleagues and companions” (Trends in Genetics, vol 29, p 1).

This theory is often called the “idiocracy” hypothesis, after the eponymous film, which imagines a future in which the social safety net has created an intellectual wasteland. Although it has some supporters, the evidence is shaky. We can’t easily estimate the intelligence of our distant ancestors, and the average IQ has in fact risen slightly in the immediate past. At the very least, “this disproves the fear that less intelligent people have more children and therefore the national intelligence will fall”, says psychologist Alan Baddeley at the University of York, UK.

In any case, such theories on the evolution of intelligence may need a radical rethink in the light of recent developments, which have led many to speculate that there are more dimensions to human thinking than IQ measures. Critics have long pointed out that IQ scores can easily be skewed by factors such as dyslexia, education and culture. “I would probably soundly fail an intelligence test devised by an 18th-century Sioux Indian,” says Nisbett. Additionally, people with scores as low as 80 can still speak multiple languages and even, in the case of one British man, engage in complex financial fraud. Conversely, high IQ is no guarantee that a person will act rationally – think of the brilliant physicists who insist that climate change is a hoax.

It was this inability to weigh up evidence and make sound decisions that so infuriated Flaubert. Unlike the French writer, however, many scientists avoid talking about stupidity per se – “the term is unscientific”, says Baddeley. However, Flaubert’s understanding that profound lapses in logic can plague the brightest minds is now getting attention. “There are intelligent people who are stupid,” says Dylan Evans, a psychologist and author who studies emotion and intelligence.

What can explain this apparent paradox? One theory comes from Daniel Kahneman, a cognitive scientist at Princeton University who won the Nobel prize in economics for his work on human behaviour. Economists used to assume that people were inherently rational, but Kahneman and his colleague Amos Tversky discovered otherwise. When we process information, they found, our brain can access two different systems. IQ tests measure only one of these, the deliberative processing that plays a key role in conscious problem-solving. Yet our default position in everyday life is to use our intuition.

To begin with, these intuitive mechanisms gave us an evolutionary advantage, offering cognitive shortcuts that help deal with information overload. They include cognitive biases such as stereotyping, confirmation bias, and resistance to ambiguity – the temptation to accept the first solution to a problem even if it is obviously not the best.

While these evolved biases, called “heuristics”, may help our thinking in certain situations, they can derail our judgement if we rely on them uncritically. For this reason, the inability to recognise or resist them is at the root of stupidity. “The brain doesn’t have a switch that says ‘I’m only going to stereotype what restaurants are like but not people’,” Ferrell says. “You have to train those muscles.”

Because it has nothing to do with your IQ, to truly understand human stupidity you need a separate test that examines our susceptibility to bias. One candidate comes from Keith Stanovich, a cognitive scientist at the University of Toronto in Canada, who is working on a rationality quotient (RQ) to assess our ability to transcend cognitive bias.

Consider the following question, which tests the ambiguity effect: Jack is looking at Anne but Anne is looking at George. Jack is married but George is not. Is a married person looking at an unmarried person? Possible answers are “yes”, “no”, or “cannot be determined”. The vast majority of people will say it “cannot be determined”, simply because it is the first answer that comes to mind – but careful deduction shows the answer is “yes”.

RQ would also measure risk intelligence, which defines our ability to calibrate the likelihood of certain probabilities. For example, we tend to overestimate our chances of winning the lottery, says Evans, and underestimate the chance of getting divorced. Poor risk intelligence can cause us to choose badly without any notion that we’re doing so.

So what determines whether you have naturally high RQ? Stanovich has found that unlike IQ, RQ isn’t down to your genes or nurture factors from your childhood. More than anything, it depends on something called metacognition, which is the ability to assess the validity of your own knowledge. People with high RQ have acquired strategies that boost this self-awareness. One simple approach would be to take your intuitive answer to a problem and consider its opposite before coming to the final decision, says Stanovich. This helps you develop keen awareness of what you know and don’t know.

But even those with naturally high RQ can be tripped up by circumstances beyond their control. “You individually can have great cognitive abilities, but your environment dictates how you have to act,” says Ferrell.

As you have probably experienced, emotional distractions can be the biggest cause of error. Feelings like grief or anxiety clutter up your working memory, leaving fewer resources for assessing the world around you. To cope, you may find yourself falling back on heuristics for an easy shortcut. Ferrell says this may also explain more persistent experiences such as “stereotype threat”. That’s the feeling of anxiety that minority groups can experience when they know their performance could be taken to confirm an existing prejudice; it has been shown time and again to damage test scores.

Perhaps nothing encourages stupidity more than the practices of certain businesses, as André Spicer and Mats Alvesson have found. Neither were interested in stupidity at the time of their discovery. Spicer, at the Cass Business School in London, and Alvesson at Lund University in Sweden, had set out to investigate how prestigious organisations manage highly intelligent people. But they soon had to tear up their thesis.

Over and over, the same pattern emerged: certain organisations – notably investment banks, PR agencies and consultancies – would hire highly qualified individuals. But instead of seeing these talents put to use, says Spicer, “we were struck by the fact that precisely the aspects they’d been trained in were immediately switched off”, a phenomenon they branded “functional stupidity”.

Their findings made sense in the context of bias and rationality. “We didn’t initially see Kahneman as the backbone to our work,” Spicer says. “But we started to notice interesting connections to the kind of things he observed in the lab.” For example, organisational practices regularly shut down the employees’ risk intelligence. “There was no direct relationship between what they did and the outcome,” says Spicer, so they had no way to judge the consequences of their actions. Corporate pressures also amplified the ambiguity bias. “In complex organisations, ambiguity is rife – and so is the desire to avoid it at all costs,” says Spicer.

The consequences may be catastrophic. In a meta-analysis last year, Spicer and Alvesson reported that functional stupidity was a direct contributor to the financial crisis (Journal of Management Studies, vol 49, p 1194). “These people were incredibly smart,” Spicer says. “They all knew that there were problems with mortgage-backed securities and structured commodities.” But not only was it was no one’s problem to look at them; the employees faced discipline if they raised their concerns, perhaps because they seemed to be undermining those with greater authority. The result is that potentially brilliant employees left logic at the office door.

The Republic of Stupidity

In light of the economic crash, the findings would seem to confirm some of Flaubert’s fears about the power of stupid people in large groups, which he referred to in jest as The Republic of Stupidity. It also confirms some of van Boxsel’s observations that stupidity is most dangerous in people with high IQ – since they are often given greater responsibility: “the more intelligent they are, the more disastrous the results of their stupidity”.

This may explain why, according to Stanovich, the financial sector has been clamouring for a good rationality test “for years”. At the moment the RQ test cannot give a definitive score, like an IQ, because you need to compare a large number of volunteers before you can develop a steadfast scale that will allow comparison between different groups of people. However, he has found that merely taking this kind of test improves our awareness of common heuristics which can help us resist their siren song. In January, he began the process of developing the test, thanks to a three-year grant from the philanthropic John Templeton Foundation.

Whether anyone will finish what Flaubert started is another question. Van Boxsel will be calling it quits after his seventh book on the topic. But the US Library of Congress has, perhaps unwittingly, taken up the baton by deciding to archive every tweet in the world.

For the rest of us, knowledge of our foolish nature could help us escape its grasp. Maybe the Renaissance philosophers, such as Erasmus, fully understood stupidity’s capacity to rule us. Below depictions of Folly, or Stultitia, you will see the acknowledgement: “Foolishness reigns in me.”

This article appeared in print under the headline “Stupid is as stupid does”

Sally Adee is a features editor at New Scientist

• From issue 2910 of New Scientist magazine, page 30-33.