Design and Evaluation of Affective Agents with Facial Expression Recognition
Master's Thesis Research | Yuan Ze University
/ 01 Overview
I designed an interactive learning system incorporating facial-expression-based affective agents and evaluated it through a study involving 40 junior-high-school students.
The research explored how emotional self-awareness and external encouragement could support students' learning experiences. It examined changes in learning achievement before and after using the system, as well as students' learning attitudes.
Details
| Organization | Yuan Ze University |
|---|---|
| Role | Graduate Researcher |
| Timeline | 2016–2018 |
| Platform | Unity / C# / Affectiva Affdex SDK |
| Methods | Study Design / Questionnaires / Quantitative Analysis / SPSS |
| Outcome | Master's Thesis / 40-Student Evaluation |
02 / Research Problem
I was initially interested in how emotion detection could be integrated into learning systems and explored several possible approaches. Through reviewing previous research, I gradually narrowed my focus to affective agents.
Previous research I reviewed on affective agents in learning environments largely focused on providing external emotional feedback, with less attention to helping students recognize their own emotional states. The Facial Feedback Hypothesis (Tomkins, 1962) provided a theoretical basis for exploring emotional self-awareness.
This motivated me to design a learning system combining two approaches: emotional self-awareness supported by a virtual avatar and external encouragement provided by a learning companion.
Research Question
What changes in learning achievement were observed, and what learning attitudes did students report after using the affective learning system?
The evaluation also examined differences associated with students’ gender and prior learning achievement.
03 / System Design
I integrated two affective agents into an English vocabulary speaking practice system. Students practiced identifying and pronouncing words, while facial-expression recognition provided signals for emotional feedback.
Preliminary Emotion-Detection Evaluation
Before the main study, I conducted a small exploratory evaluation of the emotion detection module in different gaming scenarios. By comparing manually observed facial expressions with the system's outputs, I began questioning whether all seven emotion categories provided by the SDK were necessary for learning feedback.
I later mapped the seven categories into three emotional states—positive, negative, and neutral—based on Russell's Circumplex Model.
The system comprises three main modules: the Emotion Detection Module, the Learning System Module, and the Affective Feedback Module.
Input modules
Emotion Detection Module
Facial expression recognition
Positive / Negative / Neutral
Learning System Module
Vocabulary speaking practice
Answer correctness / Accuracy
Affective Agents Module
Output components
- Self-avatarEmotional reflection
- Mood IndexAccumulated state
- Learning CompanionContextual feedback
Self-avatar — Emotional Self-awareness
A virtual avatar reflected emotional states inferred from facial expressions in real time, allowing students to observe the system's interpretation of their emotions.
Mood Index — Accumulated Emotional State
A Mood Index tracked changes based on facial expressions and answer correctness, representing accumulated emotional states throughout the learning activity.
Learning Companion — External Encouragement
The learning companion provided feedback after each answer, considering the student's current emotional state, Mood Index, answer correctness, and cumulative accuracy. I combined these factors into 36 predefined feedback scenarios.
These scenarios were defined by combining three emotional states, two Mood Index states, two answer outcomes, and three cumulative accuracy levels (3 × 2 × 2 × 3 = 36).
How Affective Feedback Was Generated
Continuous emotion monitoring
Facial Expression Detection
Continuous facial monitoring
- Positive
- Negative
- Neutral
Self-avatar
Real-time emotional reflection
Internal influence
Emotional self-awareness
Mood Index
Accumulated emotional state
Uses facial expressions and answer correctness.
After each completed answer
Answer Completed
Correct / Incorrect
Current State
- Current emotion
- Mood Index
- Answer correctness
- Cumulative accuracy
Current emotion and Mood Index come from continuous monitoring. Mood Index also incorporates answer correctness.
Learning Companion
1 of 36 predefined feedback scenarios
External influence
04 / Research Methodology
I conducted a single-session study using a one-group pretest–posttest design involving 40 seventh-grade students.
All participants completed the same three-stage procedure:
- Participants
- 40Seventh-grade students
- Total session
- 40 minPer participant
Pre-test
5 min
Learning achievement
Learning Activity
20 min
Four learning levels
Post-test & Questionnaires
15 min
Achievement + attitude questionnaires
Learning achievement was measured using pre- and post-tests. Post-study questionnaires assessed learning motivation, self-efficacy, interest, and anxiety.
I used SPSS to conduct paired-sample t-tests on learning achievement and descriptive statistical analysis of students' attitudes. The study also examined results across gender and achievement-level groups.
05 / Findings
Learning Achievement
Students demonstrated a statistically significant improvement in learning achievement after using the integrated system.
- Pre-test mean: 28.5 / 100
- Post-test mean: 44.9 / 100
- Mean increase: 16.4 points (p < .01)
Learning Attitudes
Post-study questionnaires showed relatively positive responses in learning motivation and interest. However, willingness to learn received comparatively lower ratings.
Subgroup analyses also suggested that lower-achievement students reported greater learning interest alongside higher levels of anxiety, highlighting differences in how students experienced the learning activity.
Without a control group or separate testing of each feedback mechanism, the study could not establish how much of the observed improvement was attributable to the affective agents.
06 / Reflection
This research gave me experience translating theoretical concepts into an interactive system, designing a study, collecting participant data, and conducting quantitative analysis.
Through reviewing previous studies, I realized that some of my initial ideas were supported by existing research. I enjoyed the process of exploring different possibilities and gradually narrowing them down into a more focused research direction.
Looking back, I recognize that the study design did not allow me to evaluate the effects of internal and external influence separately. If I revisited this research today, I would compare learning experiences across different conditions—with and without the Self-avatar and Learning Companion—to better understand the contribution of each feedback mechanism.