Study of emotion recognition based on fusion multi-modal bio-signal with SAE and LSTM recurrent neural network

You-jun LI; Jia-jin HUANG; Hai-yuan WANG; Ning ZHONG

doi:10.11959/j.issn.1000-436x.2017294

您当前的位置：

首页 >

文章列表页 >

Study of emotion recognition based on fusion multi-modal bio-signal with SAE and LSTM recurrent neural network

Papers | 更新时间：2024-06-05

- Study of emotion recognition based on fusion multi-modal bio-signal with SAE and LSTM recurrent neural network
- Journal on Communications Vol. 38, Issue 12, Pages: 109-120(2017)
- 作者机构：
  
  1. 北京工业大学国际WIC研究院，北京 100124
  2. 磁共振成像脑信息学北京市重点实验室，北京 100124
  3. 脑信息智慧服务北京市国际科技合作基地，北京 100124
  4. 北京未来网络科技高精尖创新中心，北京 100124
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61420106005）);The National Basic Research Program of China(2014CB744600);The International Science＆Technology Cooperation Program of China(2013DFA32180)
- DOI：10.11959/j.issn.1000-436x.2017294
  CLC： TP181，TP183
- Online First：2017-12，
  
  Published：25 December 2017
- 稿件说明：
移动端阅览
You-jun LI, Jia-jin HUANG, Hai-yuan WANG, et al. Study of emotion recognition based on fusion multi-modal bio-signal with SAE and LSTM recurrent neural network[J]. Journal on Communications, 2017, 38(12): 109-120.
DOI：

You-jun LI, Jia-jin HUANG, Hai-yuan WANG, et al. Study of emotion recognition based on fusion multi-modal bio-signal with SAE and LSTM recurrent neural network[J]. Journal on Communications, 2017, 38(12): 109-120. DOI： 10.11959/j.issn.1000-436x.2017294.

摘要

为了提高情感识别的分类准确率，提出一种将栈式自编码神经网络（SAE）和长短周期记忆单元循环神经网络（LSTM RNN）融合的多模态融合特征情感识别方法。该方法通过SAE对不同模态的生理特征进行信息融合和压缩，随后用LSTM RNN对长时间周期的融合进行情感分类识别。通过将该方法用到开源数据集中进行验证，得到情感分类准确率达到0.792 6。实验结果表明，SAE对多模态生理特征进行了有效融合，LSTM RNN能够有效地对长时间周期中的关键特征进行识别。

Abstract

In order to achieve more accurate emotion recognition accuracy from multi-modal bio-signal features，a novel method to extract and fuse the signal with the stacked auto-encoder and LSTM recurrent neural networks was proposed.The stacked auto-encoder neural network was used to compress and fuse the features.The deep LSTM recurrent neural network was employed to classify the emotion states.The results present that the fused multi-modal features provide more useful information than single-modal features.The deep LSTM recurrent neural network achieves more accurate emotion classification results than other method.The highest accuracy rate is 0.792 6

关键词

Keywords

references

聂聃 , 王晓韡 , 段若男 , 等 . 基于脑电的情绪识别研究综述 [J ] . 中国生物医学工程学报 , 2012 , 31 ( 4 ): 595 - 606 .

NIE D , WANG X H , DUAN R N , et al . A survey on EEG based emotion recognition [J ] . Journal of Biomedical Engineering , 2012 , 31 ( 4 ): 595 - 606 .

JONGHWA K , ANDRE E . Emotion recognition based on physiological changes in music listening [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2008 , 30 : 2067 - 2083 .

赵力 , 钱向民 , 邹采荣 , 等 . 语音信号中的情感识别研究 [J ] . 软件学报 , 2001 , 12 ( 7 ): 1050 - 1055 .

ZHAO L , QIAN X M , ZOU C R , et al . A study on emotional recognition in speech signal [J ] . Journal of Software , 2001 , 12 ( 7 ): 1050 - 1055 .

林奕琳 , 韦岗 , 杨康才 . 语音情感识别的研究进展 [J ] . 新能源进展 , 2007 , 12 ( 1 ): 90 - 98 .

LIN Y L , WEI G , YANG K C . A survey of emotion recognition in speech [J ] . Journal of Circuits and Systems , 2007 , 12 ( 1 ): 90 - 98 .

赵腊生 , 张强 , 魏小鹏 . 语音情感识别研究进展 [J ] . 计算机应用研究 , 2009 , 26 ( 2 ): 34 - 38 .

ZHAO L S , ZHANG Q , WEI X P . Survey on speech emotion recognition [J ] . Application Research of Computers , 2009 , 26 ( 2 ): 34 - 38 .

OTHMAN M , WAHAB A , KARIM I , et al . EEG emotion recognition based on the dimensional models of emotions [J ] . Procedia-Social and Behavioral Sciences , 2013 , 97 ( 2 ): 30 - 37 .

陈曾 , 刘光远 . 脑电信号在情感识别中的应用 [J ] . 计算机工程 , 2010 , 36 ( 9 ): 168 - 170 .

CHEN Z , LIU G Y . Application of EEG signal in emotion recognition [J ] . Computer Engineering , 2010 , 36 ( 9 ): 168 - 170 .

张栋 , 陈东伟 , 游雅 , 等 . 基于自适应Lempel-Ziv复杂度的情感脑电信号特征分析 [J ] . 计算机应用与软件 , 2014 ( 9 ): 162 - 165 .

ZHANG D , CHEN D W , YOU Y , et al . Analyzing emotional EEG signals feature based on adaptive LEMPEL-ZIV complexity [J ] . Computer Applications and Software , 2014 ( 9 ): 162 - 165 .

UPASANA T , SHYAMANTA M H . Estimation of mental fatigue during EEG based motor imagery [C ] // IHCI 2016:Intelligent Human Computer Interaction . 2016 : 122 - 132 .

BAJAJ V , PACHORI R B . Detection of human emotions using features based on the multiwavelet transform of EEG signals [M ] . Springer International Publishing , 2015 : 215 - 240 .

HOSSEINI S A , NAGHIBISISTAN M B . Emotion recognition method using entropy analysis of EEG signals [J ] . International Journal of Image Graphics＆Signal Processing , 2011 , 3 ( 5 ): 30 - 36 .

王凯明 , 钟宁 , 周海燕 . 基于改进功率谱熵的抑郁症脑电信号活跃性研究 [J ] . 物理学报 , 2014 , 63 ( 17 ): 178701 - 178701 .

WANG K M , ZHONG N , ZHOU H Y . Activity analysis of depression electroencephalogram based on modified power spectral entropy [J ] . Acta Phys Sin , 2014 , 63 ( 17 ): 178701 - 178701 .

KREIBIG S D . Autonomic nervous system activity in emotion:a review [J ] . Biological Psychology , 2010 , 84 ( 3 ): 394 - 421 .

KOELSTRA S , MUHL C , SOLEYMANI M , et al . DEAP:a database for emotion analysis; using physiological signals [J ] . IEEE Transactions on Affective Computing , 2012 , 3 ( 1 ): 18 - 31 .

PAUL E . An argument for basic emotions [J ] . Cognition and emotion , 1992 , 6 ( 3/4 ): 169 - 200 .

POSNER J , RUSSELL J A , PETERSON B S . The circumplex model of affect:an integrative approach to affective neuroscience,cognitive development,and psychopathology [J ] . Development and Psychopathology , 2005 , 17 ( 3 ): 715 - 734 .

ZHANG P , MA X , ZHANG W , et al . Multimodal fusion for sensor data using stacked autoencoders [C ] // IEEE Tenth International Conference on Intelligent Sensors,Sensor Networks and Information Processing . 2015 .

HOCHREITER S , SCHMIDHUBER J . Long short-term memory [J ] . Neural Computation , 1997 , 9 ( 8 ): 1735 - 1780 .

HUANG N E , SHEN Z , LONG S R , et al . The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis [J ] . The Royal Society , 1998 , 454 ( 1971 ): 903 - 995 .

AGRAFIOTI F , HATZINAKOS D , ANDERSON A K . ECG pattern analysis for emotion detection [J ] . IEEE Transactions on Affective Computing , 2012 , 3 ( 1 ): 102 - 115 .

BAILENSON J N , PONTIKAKIS E D , MAUSS I B , et al . Real-time classification of evoked emotions using facial feature tracking and physiological responses [J ] . International Journal of Human-Computer Studies , 2008 , 66 ( 5 ): 303 - 317 .

LISETTI C L , NASOZ F . Using noninvasive wearable computers to recognize human emotions from physiological signals [J ] . Eurasip Journal on Advances in Signal Processing , 2004 , 2004 ( 11 ): 1672 - 1687 .

FLEUREAU J , GUILLOTEL P , QUAN H T . Physiological-based affect event detector for entertainment video applications [J ] . IEEE Transactions on Affective Computing , 2012 , 3 ( 3 ): 379 - 385 .

CHUNG S Y , YOON H J . Affective classification using Bayesian classifier and supervised learning [C ] // International Conference on Control,Automation and Systems . 2012 : 1768 - 1771 .

LI X , SONG D , ZHANG P , et al . Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network [C ] // IEEE International Conference on Bioinformatics and Biomedicine . 2017 : 352 - 359 .

Views

3763

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

AI问答

⁰