03 July 2009 by Hazel Muir
SUNDAY, 1 February 2009, and 100 million Americans have got only one thing on their minds - the Super Bowl. The Pittsburgh Steelers are poised to battle the Arizona Cardinals in the most popular televised sporting event in the US. In a hotel room in New York, 46 supporters gather to watch the game, munching burgers and downing beers. Nothing strange about that, of course, aside from the machines that are monitoring these sports fans' every move and every breath they take.
The viewers are wearing vests with sensors that monitor their heart rate, movement, breathing and sweat. A market research company has kitted out the party-goers with these sensors to measure their emotional engagement with adverts during commercial breaks. Advertisers pay $3 million for a 30-second slot during the Super Bowl, so they want to be as confident as they can be that their ads are hitting home. And they are willing to pay for the knowledge. "It's a rapidly growing market - our revenues this year are four times what they were last year," says Carl Marci, CEO and chief scientist for the company running the experiment, Innerscope Research based in Boston, Massachusetts.
Innerscope's approach is the latest in a wave of ever more sophisticated emotion-sensing technologies. For years, computers in some call centres have monitored our voices so that managers can home in on what makes us fly into a wild rage. The latest technologies could soon be built into everyday gadgets to smooth our interactions with them. In-car alarms that jolt sleepy drivers awake, satnavs that sense our frustration in a traffic jam and offer alternative routes, and monitors that diagnose depression from body language are all in the pipeline. Prepare for the era of emotionally aware gadgets.
Outside of science fiction, the idea of technology that reads emotions has a brief, and chequered, past. Back in the mid-1990s, computer scientist Rosalind Picard at the Massachusetts Institute of Technology suggested pursuing this sort of research. She was greeted with scepticism. "It was such a taboo topic back then - it was seen as very undesirable, soft and irrelevant," she says.
Picard persevered, and in 1997 published a book called Affective Computing, which laid out the case that many technologies would work better if they were aware of their user's feelings. For instance, a computerised tutor could slow down its pace or give helpful suggestions if it sensed a student looking frustrated, just as a human teacher would.
She also suggested that wearable computers could sense emotion in a very direct way, by measuring your heart and breathing rate, or the changes in the skin's electrical conductance that signal emotional arousal. Wearable "mood detectors" could help people identify their stress triggers or communicate how they are feeling to others.
The most established way to analyse a person's feelings is through the tone of their voice. For several years, companies have been using "speech analytics" software that automatically monitors conversations between call-centre agents and customers. One supplier is NICE Systems, based in Ra'anana, Israel. It specialises in emotion-sensitive software and call-monitoring systems for companies and security organisations, and claims to have more than 24,000 customers worldwide, including the New York Police Department and Vodafone.
As well as scanning audio files for key words and phrases, such as a competitor's name, the software measures stress levels, as indicated by voice pitch and talking speed. Computers flag up calls in which customers appear to get angry or stressed out, perhaps because they are making a fraudulent insurance claim, or simply receiving poor service.
Voice works well when the person whose feelings you are trying to gauge is expressing themselves verbally, but that's not always the case, so several research teams are now figuring out ways of reading a person's feelings by analysing their posture and facial expressions alone.
Many groups have made impressive progress in the field, first by training computers to identify a face as such. Systems do this by searching for skin tone and using algorithms to locate features like the corners of the eyes and eyebrows, the nostrils and corners of the mouth (see diagram).
The computer can then keep track of these features as they move, often classifying the movements according to a commonly used emotion encoding system. That system recognises 44 "action units" representing facial movements. For instance, one might represent a smile - the mouth stretches horizontally and its corners go up. Add to that an eye-region movement that raises the cheeks and gives you crow's feet and now you have a beaming, genuinely happy smile rather than a stiff, polite one.
Using these techniques, computer programs can correctly recognise six basic emotions - disgust, happiness, sadness, anger, fear and surprise - more than 9 times out of 10, but only if the target face uses an exaggerated expression. Software can accurately judge more subtle, spontaneous facial expressions as "negative" or "positive" three-quarters of the time, but they cannot reliably spot spontaneous displays of the six specific emotions - yet. To accurately interpret complex, realistic emotions, computers will need extra cues, such as upper body posture and head motion.
That's because facial expressions alone are ambiguous. A smile on your face might actually signal embarrassment if it's also accompanied by a downward pitch of the head, for instance. A backward head motion is one part of an expression of disgust. But if someone combines that with a downward movement of the mouth and one raised shoulder, they're conveying indifference. "If I just looked at the face and saw the mouth going down, I would score it as sadness. But the combination with the shoulder and head motion is 'I don't care'," says Maja Pantic, who studies computer recognition of expressions at Imperial College London.
Pantic's team eventually hopes to find ways of fusing information from body gestures and facial expressions together in real time to read emotions accurately, although she concedes it may be an impossibly complex challenge. "This research is still so very new," she notes.
Basic emotions
In the meantime, they are studying the dynamics of how expressions change, to see if this can help computers identify emotions more accurately. Intuitively, most people know that a faked smile is more exaggerated than a real one, and switches on and off more abruptly. Facial-tracking technology has confirmed that, and also revealed some more subtle differences (you can see a video comparing fake and real smiles at www.newscientist.com/issue/2715).
These subtleties came to light in a 2004 study of 81 adults by Jeffrey Cohn and Karen Schmidt at the University of Pittsburgh in Pennsylvania (International Journal of Wavelets, Multiresolution and Information Processing, vol 2, p 1). They used tracking technology to compare forced smiles with spontaneous smiles provoked by comedy videos. This showed that spontaneous smiles are surprisingly complex, with multiple rises of the mouth corners.
Other teams have been highly successful at the opposite end of the emotional spectrum: pain detection. Computers are surprisingly good at distinguishing fake pain from the real thing, according to a study published this year by Gwen Littlewort of the University of California, San Diego, and colleagues.
Her team investigated whether facial expression software could distinguish people in real pain (because their hands were in iced water) from others asked to fake pain. The computer correctly classified real or fake pain 88 per cent of the time. When the team asked 170 untrained volunteers to make the judgement, they were right only 49 per cent of the time - no better than complete guesswork.
This year, Pantic and her colleagues hope to find out whether computers can accurately recognise the signs of lower back pain from facial expressions and body posture. They hope that computers might be able to distinguish between real physiological pain and the pain someone might perceive, quite genuinely, if they expect to feel pain or are depressed, but have no physiological cause for it. It could lead to more reliable ways of assessing whether painkillers are effective. "If you get a prescribed medication for acute pain, we would be able to monitor whether these medicines are actually working just by observing a person's behaviour," says Pantic.
One group of researchers has developed emotion-reading technology for a particularly vulnerable group of people. Picard and Rana el Kaliouby of MIT have built an "Interactive Social-Emotional Toolkit" (iSET), designed to help children with disorders linked to sensory processing, such as autism, to understand emotions in other people. A camera monitors the face of someone the child is talking to, and identifies 31 facial and head movements. Software interprets the combination of movements in terms of six states: agreeing, disagreeing, concentrating, thinking, interested and confused.
Then a laptop-sized screen displays six labelled bubbles that grow or shrink accordingly. If someone's nodding and smiling during the conversation, the agreeing bubble grows. If the listener looks away, a growing red bubble signals disagreement or disinterest. The team will began randomised trials of the technology this month. For 15 weeks, one group of five autistic children will use the iSET, while two control groups will use either an interactive DVD that teaches emotional awareness or have only standard classroom training. Before and afterwards, the researchers will test how well the children identify emotional expressions unaided by the iSET to see if the technology helps them learn to identify emotions for themselves.
Patronising paperclips
Not everyone welcomes these developments. William Gaver, a designer at Goldsmiths, University of London, concedes some of the applications may be beneficial, but fears emotion-sensing computers will be used in patronising ways. Who could forget Microsoft's cringe-making "paperclip" that offered help with writing letters: Microsoft wisely killed it off because people found it so irritating. But what if some emotion-triggered reincarnated "Mr Clippy" started popping up everywhere?
"The nightmare scenario is that the Microsoft paperclip starts to be associated with anything from the force with which you're typing to some sort of physiological measurement," says Gaver. "Then it pops up on your screen and says: 'Oh I'm sorry you're unhappy, would you like me to help you with that?'"
Emotion sensors could undermine personal relationships, he adds. Monitors that track elderly people in their homes, for instance, could leave them isolated. "Imagine being in a hurry to get home and wondering whether to visit an older friend on the way," says Gaver. "Wouldn't this be less likely if you had a device to reassure you not only that they were active and safe, but showing all the physiological and expressive signs of happiness as well?"
Picard raises another concern - that emotion-sensing technologies might be used covertly. Security services could use face and posture-reading systems to sense stress in people from a distance (a common indicator a person may be lying), even when they're unaware of it. Imagine if an unsavoury regime got hold of such technology and used it to identify citizens who opposed it, says Picard. There has already been progress towards stress detectors. For instance, research by Ioannis Pavlidis at the University of Houston, Texas, has shown that thermal imaging of people's faces can sense stress-induced increases in blood flow around the eyes.
His team analysed thermal videos of 39 political activists given the opportunity to commit a mock crime - stealing a cheque left in an empty corridor, made payable to an organisation they strongly opposed. They had to deny it during subsequent interrogation, and were threatened with financial penalties and punishments of loud noise if the interrogator caught them lying (empty threats at it turned out, for ethical reasons). Computer analysis of the videos correctly distinguished the 15 innocent and 24 guilty "suspects" 82 per cent of the time.
Another fledgeling technique, called laser Doppler vibrometry, measures tiny stress-related changes in respiration and heartbeat from afar - indicators that are sometimes used to gauge whether a person is stressed, and hence possibly lying.
Picard says that anyone utilising emotion-sensing systems should be obliged to gain informed consent from the people they plan to "read". At least that way, whether you find it patronising, creepy or just plain annoying, you can hit the big "off" button and it will, or at least should, leave you and your emotions in peace.
I know how you feel
DO YOU reckon you're a master of reading another's true feelings? Many people think they are, but only about 1 in 100 of us are naturally gifted at recognising emotions in someone who's trying to conceal them, says Paul Ekman, a psychologist formerly at the University of California, San Francisco.
Ekman made his name when he identified the facial expressions of the seven key emotions that are universal, regardless of nationality or culture - happiness, sadness, fear, anger, disgust, contempt and surprise. He also acts as a consultant to law-enforcement agencies, advising them on how to spot liars from clues in their facial expressions, speech and body movements.
It takes considerable effort to be a good human lie detector. To begin with, it is essential to know your subject's "baseline" behaviour when they're not stressed. Then look for deviations from this when they're under interrogation. Ekman points out that not everyone is the same. For example, some people look fearful regardless of their emotions.
So there are no absolute signs that people are definitely lying, but here are some of Ekman's top tips for spotting a fraud:
DO THEY HAVE RHYTHM?
Clues in the voice include unusually long or frequent pauses. People who are having trouble deciding exactly what to say usually use fewer hand gestures to reinforce their speech - they're less likely to "conduct" their speech by waving their hands.
LOOK OUT FOR FLICKERS
People can't help showing their true feelings for a fraction of a second. For example, a person might try to conceal their feelings of contempt, but give it away with a fleeting raised lip on one side, so look out for these micro-expressions. (Test your ability to interpret micro-expressions at http://www.facetest.notlong.com/)
SPOT THE GESTURAL SLIPS
Some gestures, called "emblems", have a precise meaning within a cultural group. Examples include a shoulder shrug with upward palms, communicating "who cares" or "I'm helpless". Usually people make them obvious, but when lying, they may display an incomplete emblem. They might rotate their hands upwards on their lap - a subconscious fragment of the shrug that betrays their feeling of helplessness at not lying well.
Hazel Muir is a freelance writer based in the UK
No comments:
Post a Comment