If hiring a babysitter, you would hire the person who seems more trustworthy.
These impressions from faces are immediate and compelling. You don’t stop to think—you form an immediate impression and make an instantaneous decision about the character of the person. And your impression and the resulting decisions can be consequential.
My coauthors and I showed study participants pairs of images of US political candidates, removing any pictures of familiar, famous people. Just using participants’ first impressions, we were able to predict election outcomes with about 70 percent accuracy, and the results have been replicated in many other countries using photos of local politicians.
University of California at Berkeley’s Gabriel S. Lenz and MIT’s Chappell Lawson, both political scientists, have followed up on this work, so we know that the effect is driven by people with little political knowledge who watch TV. They’re substituting a complex question (how competent is each candidate?) with an easy one (how competent does each candidate look?). In a sense, they’re looking for the right information in the wrong place, because it is expedient.
These impressions influence voting decisions, and many other decisions as well. As I discuss in Face Value: The Irresistible Influence of First Impressions, research has demonstrated that competent-looking CEOs are not necessarily better at their jobs, but they’re more likely to get high compensation packages. Dominant-looking cadets are more likely to achieve higher military ranks. And trustworthy-looking borrowers are more likely to get loans, and to get them with a lower interest rate.
Worryingly, there are technological startups promising to profile personalities on the sole basis of facial images. There are scientific publications claiming that algorithms can read criminal inclination, sexual orientation, political orientation, and more from facial images alone. Physiognomy has never gone away; it’s just wearing new clothes.
Discounted because of appearance
I see two reasons for why physiognomy is so appealing. First, it promises an easy solution to a complex problem, which is how to figure out the intentions and capabilities of strangers. For most of human history, people lived in extended families with rich firsthand and secondhand information about others. The emergence of modern states changed all this, and we had to learn how to live with strangers without trying to kill them or being afraid of being killed.
Most people we meet now are, thankfully, not out to kill us—but we’re still trying to figure out their intentions and capabilities. Consider an employer trying to evaluate prospective candidates for a job. In the early 20th century, the married writers Katherine M. H. Blackford and Arthur Newcomb published several influential books espousing “character analysis,” with arguments based not on Lavater’s flimsy logic but on what they understood as “evolutionary” theory. When trying to find the right person for a job, Blackford and Newcomb essentially said, just look at the candidate and do a character analysis on the basis of appearance. What the candidate said didn’t matter in their view. Letters of recommendations were worthless.
The reality is that Blackford and Newcomb were wrong. Letters of recommendation are actually more predictive than unstructured interviews. As for the idea that you learn something by analyzing a candidate at a meeting, this is called the “interview illusion.”
But appearance sways employers, even in domains such as sports with lots of evidence about the capabilities of others. The book Moneyball by Michael Lewis is about the unlikely success of Billy Beane, the general manager of the Major League Baseball team the Oakland Athletics, whose success was due to his exploiting the prejudices of appearance. He worked with much smaller revenues than other teams, yet the A’s were consistently one of the best clubhouses because he found players who were undervalued because they didn’t look the part.
Billy Beane was looking for “those guys who for their whole career had seen their accomplishments understood with an asterisk,” Lewis wrote. “The footnote at the bottom of the page said, ‘He’ll never go anywhere because he doesn’t look like a big league ball player.’ . . . Young men who failed the first test of looking good in uniform.’” Beane recognized that inferences we make from appearances are often wrong, and he used that to build a winning team, relying on statistical information based on past performance.
Making stereotypes visual
The second reason the physiognomist’s promise is so appealing is because it agrees with our intuitions. We form impressions quickly, and our impressions often seem to be shared by the people we know.
The faces you saw above were generated by computational models, which visualize our facial stereotypes. To build these models, we start with a statistical representation of faces: each face is a set of numbers. We can randomly generate thousands of faces, then ask people to rate them on perceived attractiveness, trustworthiness, or anything else of interest. In this way, we can build a computational model that captures the variations in facial appearance that lead to changes in ratings. Then, we can generate new faces (including hyperrealistic ones) and manipulate those on the basis of these models. When we show these faces to new study participants, the judgments people make match the model’s predictions.
But the faces that our models spit out do not capture personality characteristics, and this is the essential difference between the work that we do and the physiognomists’ work. The models capture systematic biases in judgments. They visualize our stereotypes or those of a particular group—in our case, mostly white American study participants. The physiognomists’ promise confuses the impression (your cognitive and emotional response) with the real thing.
The promise also assumes that faces are the same as images of those faces, when in fact images are an imperfect, variable representation. There are pictures of yourself that you like more than others, after all. Great photographers are employed precisely because they have the ability to make us look better than we usually do.
It’s not immediately obvious that image variation, even if it’s random, can generate completely different impressions of the same person. How many individuals are shown below?