The acquisition and analysis of facial expression information is the key to achieve natural communication in human computer interaction. Therefore, research on facial expression recognition has attracted considerable attention in recent years. Facial expression recognition, as an interdisciplinary research field of artificial intelligence, computer vision, image processing, psychology, cognition and so on, can realize intelligent and more natural human-computer interaction environment, which has great application value. Generally speaking, the process of facial expression recognition includes image preprocessing, face detection, facial feature location, expression feature extraction, and expression classification. Based on the existing research achievement, this book makes an in-depth study on the key issues of facial expression recognition in human-computer interaction. The main research contents are as follows:
1.Research on the constancy of color images. According to the principle of modularity in visual information processing, a fractional-step color constancy method for color images under complex illumination conditions is proposed. This method overcomes the ill-posed problem of traditional color constancy, and can maintain the color constancy of images under complex illumination, such as partial color and low illumination, and effectively improve the contrast and brightness of images.
2.Research on face detection. Face detection is the preliminary work and basis of facial expressions recognition. A face detection method based on multi-feature fusion is proposed. This method enables multiple features to cooperate with each other for efficient and accurate face detection.
3.Research on facial feature location. The accurate positioning of the facial features is the key to correct recognition. For grayscale images, an eye location method based on multi-scale sel^quotient graph and morphological filtering is proposed. For color images, a face key feature location method based on face color information features is proposed. These methods have high accuracy rate, and they are robust for the variation of the expressions and poses.
4.Research on facial expression image normalization. The geometrical characteristics and optical properties of facial expression image should be normalized before expression feature extraction. The pure expression image after normalization has uniform size, angle, and luminance, and removes the impact of the light and intensity of illumination.
5.Research on expression feature extraction and expression classifi・ cation. A facial expression recognition method based on supervised orthogonal locality preserving projection is proposed. According to the characteristics of facial expressions, this method firstly combines Gabor local statistical features with LBP (local binary patterns) texture features as composite expression features, and then reduces the feature dimension by applying supervised orthogonal locality preserving projection. Finally, facial expressions can be classified by adopting nearest neighbor algorithm. The proposed method makes use of category prior knowledge to further improve classification performance.
At last, this book summarizes the innovation points and main research results, and points out the problems of further research.
In recent years, the authors have been engaged in the research work in computer vision, image processing, pattern recognition and other fields. Combining the new achievements, new developments and trends in the field of facial expression recognition, it is necessary to write an academic work, in which the related theories and methods are systematically introduced. It is expected that the publication of this book can play a certain role in promoting the research and application in this field.
The book consists of 8 chapters, among of which Chapter 2-7 written by Song Bin from Henan University of Science and Technology, and Chapter 1 and Chapter 8 written by Zhang Zhiyong from Henan University of Science and Technology. The Publication of this book was sponsored by National Natural Science Foundation of China (Grant No.61972133 and No.12101195), Henan Province Natural Science Fund (Grant No.202300410146), Henan Province Key Scientific and Technological Projects (Grant No.222102210177, No.222102210072, No.212102210383, and No.202102210162).
During the writing process of the book, the authors refer to the books, the application results and advanced technologies of domestic and foreign counterparts. The authors are very grateful. Restricted by limited level of the writers, it is unavoidable that there are mistakes in the book. The authors urge all experts, scholars, and readers to give advice for correction.

Preface

Chapter 1 Introduction 1

1.1 Research Background and Significance 1

1.2 Overview of Facial Expression Recognition 4

1.3 Research Status at Home and Abroad 7

      1.3.1 Typical Facial Expression Recognition Algorithms 7

      1.3.2 Existing Problems in Facial Expression Recognition 13

      1.4 Research Contents 14

Chapter 2 Color Constancy Algorithm for Color Image Under Complex Illumination Conditions 20

2.1 Introduction 20

2.2 Correction of Color Deviation 21

      2.2.1 LoG Chrominance Edge Extraction 23

      2.2.2 White Balance Adjustment 28

      2.2.3 Experimental Results and Analysis 30

2.3 Luminance Enhancement 32

      2.3.1 Retinex Enhancement Algorithm 32

      2.3.2 Color Retention Enhancement Algorithm 35

      2.3.3 Experimental Results and Analysis 40

2.4 Fractional Step Color Constancy Algorithm 42

2.5 Summary of This Chapter 45

Chapter 3 Face Detection Algorithm with Multi-feature Fusion 46

3.1 Introduction 46

3.2 Skin Color Area Detection 49

      3.2.1 Common Color Space 49

      3.2.2 Skin Color Clustering Analysis 55

      3.2.3 Skin Color Area Detection Algorithm 57

3.3 AdaBoost Face Detection Combined with Skin Color Features　60

      3.3.1 AdaBoost Algorithm　62

      3.3.2 AdaBoost Face Detection Algorithm Combined with Skin Color Features 67

3.4 Face Authentication Based on Harris Corner Point Detection 69

      3.4.1 Principles for Harris Corner Point Detection 69

      3.4.2 Harris Corner Point Detection Algorithm 71

      3.4.3 Face Authentication Based on Harris Corner Point Detection 73

3.5 Experimental Results and Analysis 74

3.6 Summary of This Chapter 76

Chapter 4 Facial Features Location 78

4.1 Introduction 78

4.2 Eye Location Based on the Robustness of Illuminated Expression 79

      4.2.1 Multi-scale Self-quotient Image 80

      4.2.2 Rough Location of Morphological Filtering 81

      4.2.3 Precise Eye Location 85

      4.2.4 Experimental Results and Analysis 86

4.3 Location System of Key Color Facial Feature Point 90

      4.3.1 Overview of the System 90

      4.3.2 Eyes Location 92

      4.3.3 Mouth Location 95

      4.3.4 Experimental Results and Analysis 97

4.4 Summary of This Chapter 99

Chapter 5 Normalization of Facial Expression Image 101

5.1 Introduction 101

5.2 Rotation Normalization 102

5.3 Size Normalization 104

5.4 Illumination Normalization 106

5.5 Experimental Results and Analysis 110

5.6 Summary of This Chapter 111

Chapter 6 Extraction of Facial Expression Features 112

6.1 Introduction 112

6.2 Gabor Local Statistical Features 116

6.3 LBP Features 120

      6.3.1 Basic LBP and Its Improvement 120

      6.3.2 Uniform LBP Algorithm 122

      6.3.3 Expression Features Extracted by LBP Histogram 124

6.4 Summary of This Chapter 125

Chapter 7 Facial Expression Recognition Based on Supervised Orthogonal Locality Preserving Projection 127

7.1 Introduction 127

7.2 LPP Algorithm 129

      7.2.1 Laplacianface Algorithm 129

      7.2.2 Orthogonal Laplacianface Algorithm 133

7.3 Expression Recognition with SOLPP 135

      7.3.1 LDA 135

      7.3.2 SOLPP 136

      7.3.3 Classification of Facial Expressions 140

7.4 Experimental Results and Analysis 141

7.5 Summary of This Chapter 147

Chapter 8 Conclusion and Future Works 148

8.1 Conclusion 148

8.2 Future Works 150

References 153

中科院文献情报中心