Emotion Recognition in Human Computer Interaction Systems

1Abstract—In order to ensure comfortable life human computer interaction (HCI) systems are widely used in the smart home. It is important to detect and respond to the smart home residents’ emotional state and reduce stress levels. The HCI system for emotional state recognition is developed. This paper deals with stressful state recognition by means of the heart rate variability (HRV) analysis, because it is a noninvasive method. The emotional state should be identified in the situations which correspond to real life at home: a person sits, walks, and changes his/her posture over time. The impact of the emotional state and the posture impact on heart rate variability are examined. Time domain, frequency domain and nonlinear parameters are calculated. The parameters that are most sensitive to the emotional state are chosen. Variability of the HRV parameters are verified over time. It was found that posture has a great impact on the HRV parameters, so posture detection subsystem is integrated in our HCI system for emotion recognition. The subjectdependent thresholds should be used for emotional state recognition.

parameters are commonly used.In all related works the emotional state is determined when the subject's posture is static.Our contribution is development of HCI system allowing the emotional state recognition when the posture is dynamic (changes over time).

II. BACKGROUND AND RELATED WORKS
Before designing the automatic stress recognition system we have to deal with three problems: determine which parameters best reflect the human emotional state; detect the emotional state when the posture changes, decide how to cause stress during the experiment by using non-violent methods.
The emotional state of a human may be detected by measuring physiological parameters or by reading the body language (face, eyes, and gestures).The body languagebased methods are not very effective because masking of emotions is common in everyday life (the term, such as "poker face") [1].Physiological signals are measured by using sensors.Two conditions must be satisfied: the sensors must be unobtrusive, so that natural behaviour is not sacrificed; the finalised data stream must work in real time based on the processing power of an average computer [2].
That is why HRV measures were chosen for emotional state recognition.

A. HRV Parameters
The raw electrocardiogram data and automatic detection of the QRS-complexes are used for calculating the RR intervals (interbeat intervals -IBI).The HRV parameters (time domain, frequency domain and nonlinear) are calculated.
Some studies have attempted to investigate the stress state by using only frequency domain parameters [3], [4].The frequency and time domain parameters are used in other works [5]- [7].The results were obtained by the calculation of power spectral density (PSD), and all participants in the experiment did not change their posture.
For estimating PSD the RR interval series representing the unevenly sampled data sequence are used.If the FFT is used to calculate the PSD, you should resample the data at uniform intervals.A linear or cubic spline interpolation is used for resampling.Due to the re-sampling operation the power spectrum is distorted.The spectral peaks are displaced towards the low frequency region and their values are attenuated [8].The Lomb periodogram method for estimating the power spectral density of the RR interval series does not require re-sampling and is used for the analysis for HRV signals.In this case the spectral peaks are not displaced towards the low frequency region but their values are attenuated.Most frequency domain measures are especially susceptible to the outliers, particularly LF and HF power can be > 1000 % error [9].Some studies identify stressful emotional states by using only the time domain parameters of HRV [10].The participants of the experiment were both sitting and walking, but the stress state was determined when they were sitting.Walking participants were not in a stressful state.
The Poincare Plot and the Approximate Entropy are widely used to analyse nonlinear properties of HRV [11].The Poincare Plot is a graphical representation of the correlation between successive RR intervals, for instance, the plot of RRj+1 versus RRj.The Poincare plot is used to calculate the standard deviation of the point perpendicular to and along the line-of-identity referred to as SD1 and SD2, respectively [12].

B. The Impact of Music on the Emotional State
A big issue in our research was how to gather data corresponding to a real stress state.Many reports investigate the potential of audio and visual patterns as reliable tools for emotion recognition.
A method has been developed for a user -dependent emotion recognition based on the processing of physiological signals [13].In this research, the database used for emotion recognition is obtained from multiple subjects when they were experiencing specific feelings.To arouse the emotional state of the subject, film clips were used as stimulus.Four physiological signals, electrocardiogram (ECG), skin temperature (SKT), skin conductance (SC) and respiration were selected to extract 22 features for recognition.According to the authors, the research indicated the future feasibility of user-independent emotion recognition by using physiological signals.
Emotional quality level recognition based on HRV was performed [14].Two films to evoke the subjects' level of joy and sad emotion were provided.It was found that subjects of different characteristics reveal different bio-signal responses.All the subjects have been classified as optimistic or pessimistic based on self-scaling reports.The optimistic subjects have higher response to positive films; the pessimistic subjects have higher response to negative films.
Heart rate variability with repetitive exposure to music has been investigated [15].The participants were exposed to three conditions: sedative music (melodious, delicate, harmonic, and romantic), exciting music (loud, dynamic, and rhythmic, elicited tension and excitement), and no music at all.The present study examined the changes in subjective and psychophysiological responses to sedative and exciting music.Differences among the three experimental conditions were shown in changes in HRV.The LF component and the LF/HF ratio increased during EM and SM, but decreased during NM.
The user-independent emotion recognition system was presented [16].The authors used multimodal stimuli (audio, visual, and cognitive) to cause targeted emotions (sadness, stress, anger, and surprise) from 175 children aged five to eight.The classification ratio of 78 percent was achieved for three emotions (sadness, stress, and anger) and the ratio of 62 percent were achieved for four emotions (sadness, stress, anger, and surprise) by adopting the support vector machines as a pattern classifier.The effectiveness was determined from the self-report results, where the subjects were requested to give a verbal rating of the emotion.
The potential of physiological signals as reliable channels for emotion recognition was investigated and automatic recognition systems have been proposed [17].In order to collect a physiological data set, the authors used a musical induction method.Four-channel biosensors were used to measure electromyogram, electrocardiogram, skin conductivity, and respiration changes.The classification of four musical emotions (positive/high arousal, negative/high arousal, negative/low arousal, and positive/low arousal) was performed by using the extended linear discriminant analysis.Two common scales of emotions are valence and arousal.Valence represents the pleasantness of stimuli and arousal represents the level of activation.
It can be stated that during all the experiments the subjects stayed in the same posture and sat still.Practically all the experiment participants rested prior to the measurement of HRV parameters.The influence of light physical load (for example, going upstairs, and downstairs) was not investigated.In the case when the emotional state recognition system is used in a smart home, the subjects will change their posture and walk.A smart home resident will be rested in the morning and will have received some physical load in the evening.The HRV parameters will be different in the morning and in the evening.It is necessary to note that the value of the parameter changes over time.
The novelty of the work is that it is necessary to detect the subject's emotional state when the subject changes his/her posture and moves.Also, the change of the HRV parameters over time was evaluated.

III. EXPERIMENT
For investigation of the emotional state, HRV was measured because it is a non-invasive method widely used in various psychological and medical researches.Other physiological signals (skin temperature, skin conductance, respiration) were not used because these signals are useful in clinical research, but it is uncomfortable to constantly wear a lot of sensors at home.The time domain, frequency domain and nonlinear parameters were extracted from HRV.

A. Materials and Design
The HCI system for emotional state recognition was developed, when posture is dynamic.The hardware architecture of our system is represented in Fig. 1.The system is composed of body area sensors network and a human computer interface.The wearable sensors are used.The ECG recorder and transmitter are mounted in the chest band.
A portable ECG recorder consisting of an amplifier STM32373C-EVAL evaluation board and a 16-bit ADC in the STM32F373VCT6 microcontroller with 1 kHz sampling rate was used.The main difference between the similar systems and our system is the integrated posture detection system.The posture detection system in a smart home is used for other purposes, but can be used for the smart home resident's emotional state recognition.This homemade system like other similar systems [18] uses triaxial integrated accelerometers and wireless transmitters.
It is very important to handle artefacts in the interbeat intervals, because they may distort HRV parameters.Artefact (sometimes artifact) -a product or effect that is not present in the natural state (of an organism, etc.) but occurs during or as a result of investigation or is brought about by some extraneous agency such as electrode movement.A software tool ARTiiFACT was used for ECG processing [19].This software allows extracting interbeat intervals (IBI), detecting and correcting artefacts caused by inaccurate row data, hardware failures, and motions and calculating time domain and frequency domain parameters.
This experiment was designed to detect stress by using heart rate variability (HRV) parameters measured in a real home environment.Therefore, some standard clinical requirements (no movement, no coffee past 24 h and so on) were not applied to the participants.

B. Participants
The subjects were 49 computer science students of Kaunas University of Technology.Among the participants were 45 men and 4 women, the average age 21 years.They reported that they did not have cardiovascular disease, did not use drugs and were healthy on the day of testing.They were not asked to give up smoking, coffee and alcoholic beverages prior to the experiment due to the specific nature of the research.All the participants in the experiment were asked to provide data on smoking, exercise (past 24 h), caffeine (past 24 h), and alcohol (past 12 h).The figures calculated are the following: smokers -14 %, exercise -22 %, caffeine -24 %, alcohol -14 %.

C. Experimental Conditions
The aim of the experiment is to measure HRV from the participants in four real domestic situations: relaxing (R), mental stress (MS), slight physical load (PL) and mental stress combined with physical load (MSPL).
The relaxing state was modelled by the relaxing music and the sitting posture.Mental stress was simulated by using the most irritating music -noise music, i.e. music that employed noise as a musical resource.Physical load should simulate typical behaviour at home -standing or walking around in the room.
The experiments were conducted in a soundproofed room individually for each subject.The participant was asked to wear headphones while the experiments proceeded.The experimental conditions were changed over time as represented in Fig. 2. In order to simulate mental stress (relax), two opposite types of music were used: a frustrating music (negative valence, high arousal) and a relaxing music (positive valence, low arousal).Relaxing music denotes blissful, pleasurable, and tender music, while frustrating -noisy, loud, and irritating music.The musical excerpts were all instrumental.The sample of relaxing music was Ludwig van Beethoven's "Fur Elise", and the sample of frustrating music -mini CD "Beauty Beast" from the apologist of Japanese noisy music Masonna.
The participants were asked to evaluate the level of stress caused by frustrating music in the 10 points scale.The final grading results: minimum -2, maximum -10, average -6 or 7.

IV. METHODS
Artefacts may be processed in three ways: 1) deletion, 2) estimation by using linear interpolation, 3) estimation by using cubic spline interpolation.Deletion prevents emergence of artefact interbeat intervals, but changes the length of data set and may bias the data.Interpolation does not change the length of the data set, however, there is a possibility to estimate erroneously the length of the inserted interbeat intervals.Commonly used time-domain parameters, such as SDNN, RMSSD, NN50, pNN50 [20] are calculated from the IBI time series.Time parameters are depicted in Table I.

B. Frequency Domain Parameters
The frequency domain parameters are calculated using the Fast Fourier Transformation (FFT).A cubic spline interpolation is used for resampling.FFT uses a Hanning window and interpolation rate of 4 Hz.The frequency domain parameters are depicted in Table II.

C. Nonlinear Parameters
The two basic nonlinear parameters are SD1 and SD2 -so called Poincare plot descriptors [21].
The entire interbeat intervals time series RR ≡ (RR1, RR2, …, RRn+1), derived from an electrocardiogram are used for calculation SD1 and SD2: where It is suggested that index SD1 is a measure of short-term heart rate variability and index SD2 is a measure of longterm heart variability.These indexes are measures of the heart variability over a single beat [21].A software tool ARTiiFACT does not calculate indexes SD1 and SD2.MATLAB was used for calculation SD1 and SD2 parameters from the IBI series [20].

V. RESULTS
We collected electrocardiograms from 49 subjects under four conditions: R, PL, MS and MSPL.Then the time domain, the frequency domain and the nonlinear parameters of HRV were calculated.Primarily, frequency domain parameters were examined.
Parameter LF/HF of five participants under different conditions is presented in Table III.
The Table III indicates that the frequency domain parameters more reflect the human posture (sitting or walking) than the emotional state.The value of parameter LF/HF also increases in stressful situations, but less than when the posture is changed.There are some exceptions.
Parameter LF/HF decreased in a stressful state when the third subject was walking.Also this parameter decreased in a stressful state when the third subject was sitting.The mean LF/HF values of all experiment participants are represented in Fig. 3.This figure represents variation of the parameter of a participant who was training on the previous evening and was drinking coffee in the morning.The experimental data showed that the frequency domain parameters are very sensitive to coffee, alcohol (even small doses on the eve of the experiment) and a physical load.The values of frequency domain parameters HF and LF are highly dependent on the artefacts processing (deletion, linear or cubic spline interpolation).It was found that the parameter values can change by up to 30 percent depending on the method of artefacts processing The frequency domain ranges of parameter values are highly subject -dependent, so it was decided not to use the frequency domain parameters for emotional state recognition.
In order to compare the dependence of parameters on the emotional state relative average values of the time domain parameters were calculated: Parameters are presented in Fig. 4. The values of parameters SDNN and pNN50 increased in a stressful state, and decreased while walking.The value of parameter RMSSD increased in a stressful state and marginally changed while walking.The value of parameter MEANHR decreased in a stressful state and increased while walking.The main problem with using the following parameters for finding emotional state is that their values change with time.Fig. 5 presents time domain parameters of the same person that were measured at different times.Time domain parameters of 31 participants were measured twice, 10 participants were measured four times and 8 participants six times.Every time the same person's parameters differed.Another problem is that the absolute value of time domain parameters are subject -dependent and it is impossible to set the absolute threshold values for emotional state recognition.For example, in [10] it is suggested that parameter HR indicates stress when its level exceeds 86.The experiment revealed that in the relax state this parameter ranged between 67 and 96.
Nonlinear parameters were calculated from the collected ECG.The analysis revealed that the shape of Poincare plot more reflects human posture, rather than the emotional state.Fig. presents Poincare plots under four different conditions: "relax sitting", "relax walking", "stress sitting", and "stress walking".
Nonlinear parameters SD1 and SD2 evaluate the width and the length of Poincare plot.In [10], these parameters are used to determine the emotional state of the human.It is suggested that in the case when SD2 is lower than 64.6 ms, the ECG record must be classified as under a stress, otherwise -as in a condition of rest.The experiment revealed that SD2 value of 40 % participants was smaller than 64.6 ms when they were walking and listening to sedative music.Fig. 7 presents SD1 and SD2 mean values under four different conditions.The data presented in Fig. 7 confirms that the nonlinear parameters depend more on the physical load than on the emotional state and it is impossible to use SD2 threshold 64.6 ms when the posture is changed.Five parameters: MEANHR, SDNN, RMSD, pN50, and SD1 were chosen for the classification of the emotional state.Parameter SD2 was not used because it is highly correlated with SDNN (correlation coefficient 0.99) [22].Our goal is to classify ECG recordings as a stressful or not a stressful state.The ECG analysis showed that HRV are highly dependent not only on the emotional state, but also on the human posture (he/she sits or walks).The emotional state should be classified in four classes: sitting -stress, sittingno stress, walking -stress, walking -no stress.Because the range of the variation of parameters overlap in different emotional states, it is difficult to make a decision (for example, sitting in a stressful state, or walking in a nonstressful state).It is planned to identify the human's posture in a smart house, so we use the sixth additional parameter of a human posture (sitting, standing or walking).Different thresholds of parameters were used for identification the human emotional state when the person was sitting and

Fig. 4 .
Fig. 4. The relative average values of time domain parameters.

Fig. 5 .
Time domain parameters of subject X: a -the first experiment; bthe second experiment.

TABLE I .
TIME DOMAIN MEASURES OF HRV.

TABLE II .
FREQUENCY DOMAIN MEASURES OF HRV.

TABLE III .
FREQUENCY DOMAIN PARAMETER LF/HF.