Facial Coding Research

What is facial coding research?

Have you ever used a smiley face in a text message? It’s an easy way to communicate a simple emotion. When we look at a smiling face, we understand what is being communicated even in the absence of verbal communication.

Interpreting facial expressions and other nonverbal communications through facial coding research is an increasingly popular marketing and research tool for gauging consumer sentiment.

Not all emotions are as intuitive to understand as a smiley face. Researchers use the science of facial coding to accurately decode the mixed signals hiding in many facial expressions. Here we provide an overview of facial coding and explain how it might benefit your business.

The face is the key to a lot of human emotions

Understanding what your customers like and dislike can be critical to business success. Sometimes this is about getting a reaction on what they buy now, on other occasions it can be about testing ideas and getting inspiration for possible future business direction. Either way, finding a way to capture and decode their emotions and feelings can be a helpful tool to improve the success of your business, whether it be in product development, marketing communication or user experience.

In everyday life, we rely on people’s facial expressions. From stepping into the driveway in the morning and seeing the smile of your neighbor, to waiting in line at the pharmacy and watching the concentration of the pharmacist dealing with a patient, most of us believe that we can read into any given situation based on people’s facial expressions.

However, for some reason, when it comes to market research, we often feel hesitant about relying on facial expressions. They can seem to be unscientific or misleading, you may have experienced this yourself watching a focus group, where respondents smile at a concept shown to them, but upon more in-depth discussion, it transpires that in fact, they do not have the positive feelings towards it that their initial smiles may have seemed to suggest.

The human face has developed over many thousands of years to be one of the primary communication tools used by humans. It has evolved in a way which allows even subtle movements and gesture to communicate thousands of different messages. At the same time, the human brain has developed to interpret these messages seen in other people’s faces. So, although facial expressions and feelings may not always align, there is a scientific approach to understanding facial expressions. Facial coding is a technique to allow researchers to unlock the messages people are sending with their faces, whether they mean to expose those feelings or not.

The facial nerve connects most of the muscles in the face with the brain. It is the same part of our brain stem that triggers our facial expressions and controls emotional processing and regulation.

Facial coding translates facial expressions into quantifiable judgments

The science behind facial coding is relatively simple. Based on a database of how people’s faces look and how they feel, scientists have developed an interpretative code of significant facial expressions.

If you have read an old spy thriller where the hero relies on a codebook to decipher messages, then you already understand how facial coding works. Certain facial expressions are coded as meaning certain things. By analyzing the look on respondents’ faces and matching them to the codebook, researchers can form an understanding of how the respondents feel. Based on an extensive scientific database of coded expressions, the readout tends to be highly accurate.

In practice, this can be simple to implement. For example, let’s imagine that a retailer has a set of new store designs that they are considering bringing to market. By showing the designs to shoppers and capturing their immediate facial reactions, they have the raw data necessary for facial coding. This data can immediately be translated into a set of emotions which indicate the extent to which a particular design is liked or disliked, as well as what other feelings it evokes.

On the one hand, this might be seen to be an alternative to qualitative research. Whereas qualitative research involves much interpretation about the relationship between what consumers say and what they actually feel, facial coding research is a cut and dry read on their emotions based on their expressions. However, facial coding research and qualitative work can complement each other. Facial coding can provide a readout of how respondents feel about a particular stimulus presented to them. The facial coding data can provide a focus for more nuanced qualitative work. Knowing what they feel, the qualitative questioning and observation can zoom in on understanding why they feel that way.

Facial coding can work in tandem with research using broader body language cues. It’s often preferred over broader body language research due to its efficiency. As the old proverb says, the eyes are the window to the soul: it is the face that expresses much of the human emotional condition.

“People don’t think their feelings, they feel them. So facial coding offers the huge advantage of capturing and quantifying the emotional responses that tend to drive behavior at the same time that it helps resolve the risk of lip-service answers from study participants. Manual facial coding (our specialty) benefits greatly from a trained observer, given accuracy levels of 90%+ versus much lower levels at this point still for software-based approaches to facial coding. Customized analysis, and the flexibility of using facial coding in conjunction with verbal input from participants are other strengths we offer.” Sensory Logic

Dan HillPresident, Sensory Logic Inc.

The face expresses feelings at different levels

When people talk about the face in everyday life, they typically use the word “expressions.” These are the broad set of looks the face can adopt to communicate a given feeling. Expressions tend to operate at quite a high level, such as happiness, sadness, confusion, dislike or anger.

These can certainly be helpful in marketing research. If facial coding analysis were only able to differentiate between the expressions for liking and dislike, it would already be a powerful tool in a market researcher’s arsenal of methodologies. However, it can do better than that.

The next level of facial coding looks at what is known as micro-expressions. These are smaller elements which taken together ladder up into an expression. A simple way to understand this is to think about a series of high-level emotions, such as anger, liking or incomprehension. Under each of those, think of specific indications which feed into the display of such a sentiment.

Take incomprehension as an example. How do you know that someone is expressing incomprehension on his or her face? They may have raised eyebrows, pursed lips, a frown on their forehead and all manner of other individual signs that collectively add up to what you and most people would understand as a look of incomprehension. In layman’s terms, this is an example of the concept of micro-expressions.

While in this example, the micro-expressions all support a single expression, in real life research they may have a more complicated role. Sometimes, a respondent will simultaneously display micro-expressions which are conflicting. For example, they may show indicators of excitement and skepticism at the same time. Rather than reducing the utility of facial coding research, this fact serves to highlight its potential power. Facial coding research allows market researchers to understand more than just the high-level expressions a typical person could probably see for themselves.

Facial coding research also provides a scientifically coded snapshot of the full range of micro-expressions on a respondent’s face. These can be various and not obviously connected to the overall expression, for example when a respondent has one feeling, but there are also elements of doubt in that feeling.

The micro-expressions can also often be so subtle that they would not be evident to the untrained eye. However, facial coding research can capture the full gamut of facial expressions in real-time and decode them for their meaning.

Facial Expression Analysis Techniques

There are many different facial expressions that can be collected and analyzed in different ways. Each approach has different pros and cons. There are three main approaches.

Tracking of facial electromyographic activity (fEMG). This is a very accurate method where the activity of facial muscles is tracked with electrodes attached to the skin surface. fEMG detects and amplifies the tiny electrical impulses generated by the facial muscles as they contract. The pro’s of fEMG is that it is a non-invasive, precise way to continuously measure facial muscle activity. It is very accurate and can measure even subtle facial muscle activity even when respondents are instructed to inhibit their emotional expressions. The cons are that it requires electrodes, cables and amplifiers and therefore heightens a respondent’s awareness of what is being measured. While precise, it is also very sensitive and must be conducted in a very controlled environment. It also requires expert biosensor processing skills.
The second approach for Facial Coding is the Facial Action Coding System (FACS). This is achieved through live observation and a manual coding system that has a standardized classification system of facial expressions. FACS is conducted by expert facial coders. This approach allows for coding of macroexpressions which are typically obvious to the naked eye as well as microexpressions which typically last less than half a second and occur when trying to repress the current emotional state. These subtle expressions are generally associated with emotions that have a lower intensity. Some of the benefits of FACS is it is a non-intrusive, objective, reliable method of measuring facial expressions. FACS also effectively measures intensity of facial coding. The limitations of FACS is that is relies on well trained experts.
The third approach for facial coding is by automatic facial expression analysis using computer vision algorithms. These technologies use cameras embedded in laptops and other devices to capture videos of respondents as they are exposed to content. Each software has different metrics that they track but typically they measure 7 basic emotions and the valence of them. One of the big advantages of Facial Coding software is that this automatic coding is accomplished objectively versus manual coding. Automatic coding can be done both online and offline where respondents are recorded and then video is submitted for coding. Automatic coding is cost effective and quick. One of the major negatives of online facial coding is respondents being miscoded due to yawning or eating while being recorded. These observations should be removed from any conclusions that are being made from the analysis.

Facial coding works best on binary-style questions

We have already seen that the face has a vast range of expressions. Sometimes it will express multiple things at one time. You know that from your own experience, for example at a time when you hear news which makes you both happy and sad simultaneously. An example would be that a colleague you like is leaving your company because she has been offered a higher position at a different firm.

Although these multi-faceted facial expressions can be coded and interpreted, they do raise a problem of interpretation. If you show a consumer some stimulus and they respond with multiple, possibly conflicting micro-expressions at the same time, what do you draw from that? You are left to make interpretative leaps or engage in qualitative research, which undermines some of the benefits of facial coding in the first place.

To overcome this, the design of facial coding research is critical. An effective design approach will express the information the study seeks to elicit in a series of binary questions. The use of binary questions means that the range of possible answers to a question is likely to be limited to two distinct groups. This makes the interpretation clear and far less open to ambiguity.

It is also a relatively straightforward approach. It simply means designing the series of questions and stimuli as a sort of breadcrumb trail, where each leads on from one another and perhaps elicits only a limited amount of information. Whereas in qualitative research we often hear of the importance of open-ended questions, in this binary approach, there can be value in closed questions which elicit a clear answer facially which is easy to interpret.

Facial Coding can help improve advertising effectiveness, and much more

One of the most common applications for facial coding research is copy testing. When showing respondents commercials, facial coding research can be used to gauge their response in terms of understanding and copy appeal. Where the copy is not static but goes on for some time, for example with a television commercial or video, the research also allows for a detailed understanding second by second of how the reactions change. This timeline-based approach can be useful to help improve copy where parts are currently working, but other parts seem to elicit confusion or low engagement, signaling that they need to be improved.

The methodology has application beyond merely copy testing. It can also be used to gauge reaction to things such as store design, product design, branding, communication, and new product ideas.

As it can provide a numerical readout on reaction, facial coding can be particularly helpful in assessing how you are performing versus your competitors. Showing your brand or communication and then those of competitors to the same respondents will give you a clear comparison of how consumers react to them differently.

Facial Coding is an affordable, accessible technology

Facial coding has legitimacy and utility which makes it attractive to many users as a research technique.

However, it offers an additional benefit in today’s digital environment. It can often be done at a highly competitive cost. As the technology merely requires an image of facial reaction and an interpretative tool to match elements of the expression against a coded database of expressions, it is a natural fit for a simple version of artificial intelligence. Consumers with a webcam can participate in a panel without having to leave the comfort of their own homes. In this approach, the researcher shows the stimulus to the respondent via their home computer, which works well for television commercials or other video content. For product communication or concept ideas, the stimulus needs to be designed thoughtfully so that it can be presented in this environment without the need for additional explanatory information or context setting.

Interestingly, facial expressions are so hardwired into the human behavioral system that they are consistent even across different demographic groups. This consistency across demographic groups is another attraction of facial coding using digital technology. In some research techniques, it is necessary to design different research methodologies dependent on the age, ethnicity or socioeconomic demographics of the target respondent group. Facial coding research, by contrast, works consistently across different groups in the same way.