Real vs. immersive-virtual emotional experience: Analysis of psycho-physiological patterns in a free exploration of an art museum

English version

Autoři: Javier Marín-Morales ^aff001; Juan Luis Higuera-Trujillo ^aff001; Alberto Greco ^aff002; Jaime Guixeres ^aff001; Carmen Llinares ^aff001; Claudio Gentili ^aff003; Enzo Pasquale Scilingo ^aff002; Mariano Alcañiz ^aff001; Gaetano Valenza ^aff002
Působiště autorů: Instituto de Investigación e Innovación en Bioingeniería, Universitat Politècnica de València, València, Spain ^aff001; Bioengineering and Robotics Research Centre E Piaggio & Department of Information Engineering, University of Pisa, Pisa, Italy ^aff002; Department of General Psychology, University of Padua, Padua, Italy ^aff003
Vyšlo v časopise: PLoS ONE 14(10)
Kategorie: Research Article
doi: https://doi.org/10.1371/journal.pone.0223881

Souhrn

Virtual reality is a powerful tool in human behaviour research. However, few studies compare its capacity to evoke the same emotional responses as in real scenarios. This study investigates psycho-physiological patterns evoked during the free exploration of an art museum and the museum virtualized through a 3D immersive virtual environment (IVE). An exploratory study involving 60 participants was performed, recording electroencephalographic and electrocardiographic signals using wearable devices. The real vs. virtual psychological comparison was performed using self-assessment emotional response tests, whereas the physiological comparison was performed through Support Vector Machine algorithms, endowed with an effective feature selection procedure for a set of state-of-the-art metrics quantifying cardiovascular and brain linear and nonlinear dynamics. We included an initial calibration phase, using standardized 2D and 360° emotional stimuli, to increase the accuracy of the model. The self-assessments of the physical and virtual museum support the use of IVEs in emotion research. The 2-class (high/low) system accuracy was 71.52% and 77.08% along the arousal and valence dimension, respectively, in the physical museum, and 75.00% and 71.08% in the virtual museum. The previously presented 360° stimuli contributed to increasing the accuracy in the virtual museum. Also, the real vs. virtual classifier accuracy was 95.27%, using only EEG mean phase coherency features, which demonstrates the high involvement of brain synchronization in emotional virtual reality processes. These findings provide an important contribution at a methodological level and to scientific knowledge, which will effectively guide future emotion elicitation and recognition systems using virtual reality.

Klíčová slova:

Central nervous system – Electroencephalography – Emotions – Eyes – principal component analysis – Psychometrics – Support vector machines – Virtual reality

Zdroje

1. Picard RW. Affective computing. MIT press; 1997.

2. Picard RW. Affective Computing: Challenges. Int J Hum Comput Stud. 2003;59: 55–64. doi: 10.1007/BF01238028

3. Jerritta S, Murugappan M, Nagarajan R, Wan K. Physiological signals based human emotion Recognition: a review. Signal Process its Appl (CSPA), 2011 IEEE 7th Int Colloq. 2011; 410–415. doi: 10.1109/CSPA.2011.5759912

4. Harms MB, Martin A, Wallace GL. Facial emotion recognition in autism spectrum disorders: A review of behavioral and neuroimaging studies. Neuropsychol Rev. 2010;20: 290–322. doi: 10.1007/s11065-010-9138-6 20809200

5. Lindal PJ, Hartig T. Architectural variation, building height, and the restorative quality of urban residential streetscapes. J Environ Psychol. Elsevier Ltd; 2013;33: 26–36. doi: 10.1016/j.jenvp.2012.09.003

6. Ekman P. Basic Emotions. Handbook of cognition and emotion. 1999. pp. 45–60. doi: 10.1017/S0140525X0800349X

7. Barrett LF. The theory of constructed emotion: an active inference account of interoception and categorization. Soc Cogn Affect Neurosci. 2017;12: 1–23. doi: 10.1093/scan/nsw154 27798257

8. Russell J. A., & Mehrabian A. Evidence for a three-factor theory of emotions. J Res Pers. 1977;11: 273–294.

9. Calvo RA, D’Mello S. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput. 2010;1: 18–37. doi: 10.1109/T-AFFC.2010.1

10. Valenza G, Greco A, Gentili C, Lanata A, Sebastiani L, Menicucci D, et al. Combining electroencephalographic activity and instantaneous heart rate for assessing brain–heart dynamics during visual emotional elicitation in healthy subjects. Philos Trans R Soc A Math Phys Eng Sci. 2016;374: 20150176. doi: 10.1098/rsta.2015.0176 27044990

11. Valenza G, Lanata A, Scilingo EP. The role of nonlinear dynamics in affective valence and arousal recognition. IEEE Trans Affect Comput. 2012;3: 237–249. doi: 10.1109/T-AFFC.2011.30

12. Valenza G, Nardelli M, Lanatà A, Gentili C, Bertschy G, Paradiso R, et al. Wearable monitoring for mood recognition in bipolar disorder based on history-dependent long-term heart rate variability analysis. IEEE J Biomed Heal Informatics. 2014;18: 1625–1635. doi: 10.1109/JBHI.2013.2290382 24240031

13. Marín-Morales J, Higuera-Trujillo JL, Greco A, Guixeres J, Llinares C, Scilingo EP, et al. Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors. Sci Rep. 2018;8: 13657. doi: 10.1038/s41598-018-32063-4 30209261

14. He C, Yao Y, Ye X. An Emotion Recognition System Based on Physiological Signals Obtained by Wearable Sensors. In: Yang C, Virk GS, Yang H, editors. Wearable Sensors and Robots: Proceedings of International Conference on Wearable Sensors and Robots 2015. Singapore: Springer Singapore; 2017. pp. 15–25. doi: 10.1007/978-981-10-2404-7_2

15. Nakisa B, Rastgoo MN, Tjondronegoro D, Chandran V. Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors. Expert Syst Appl. Elsevier Ltd; 2018;93: 143–155. doi: 10.1016/j.eswa.2017.09.062

16. Kory Jacqueline D, Sidney. Affect Elicitation for A ffective Computing. The Oxford Handbook of Affective Computing. 2014. pp. 371–383.

17. Ekman P. The directed facial action task. Handbook of emotion elicitation and assessment. 2007. pp. 47–53.

18. Harmon-Jones E, Amodio DM, Zinner LR. Social psychological methods of emotion elicitation. Handb Emot elicitation Assess. 2007; 91–105. doi: 10.2224/sbp.2007.35.7.863

19. Roberts N a., Tsai JL, Coan J a. Emotion elicitation using dyadic interaction task. Handbook of Emotion Elicitation and Assessment. 2007. pp. 106–123.

20. Baños RM, Botella C, Alcañiz M, Liaño V, Guerrero B, Rey B. Immersion and Emotion: Their Impact on the Sense of Presence. CyberPsychology Behav. 2004;7: 734–741. doi: 10.1089/cpb.2004.7.734 15687809

21. Lange E. The limits of realism: Perceptions of virtual landscapes. Landsc Urban Plan. 2001;54: 163–182. doi: 10.1016/S0169-2046(01)00134-7

22. Baños, R. M., Liaño, V., Botella, C., Alcañiz, M., Guerrero, B., & Rey B. Changing induced moods via virtual reality. In: Springer, Berlin H, editor. International Conference on Persuasive Technology. 2006. pp. 7–15. doi: 10.1007/11755494_3

23. Peperkorn HM, Alpers GW, Mühlberger A. Triggers of fear: Perceptual cues versus conceptual information in spider phobia. J Clin Psychol. 2014;70: 704–714. doi: 10.1002/jclp.22057 24353196

24. Meehan M, Razzaque S, Insko B, Whitton M, Brooks FP. Review of four studies on the use of physiological reaction as a measure of presence in stressful virtual environments. Appl Psychophysiol Biofeedback. 2005;30: 239–258. doi: 10.1007/s10484-005-6381-3 16167189

25. Higuera-Trujillo JL, López-Tarruella Maldonado J, Llinares Millán C. Psychological and physiological human responses to simulated and real environments: A comparison between Photographs, 360° Panoramas, and Virtual Reality. Appl Ergon. 2016;65: 398–409. doi: 10.1016/j.apergo.2017.05.006 28601190

26. Egan D, Brennan S, Barrett J, Qiao Y, Timmerer C, Murray N. An evaluation of Heart Rate and Electrodermal Activity as an Objective QoE Evaluation method for Immersive Virtual Reality Environments. 2016; 3–8. doi: 10.1109/QoMEX.2016.7498964

27. Bian Y, Yang C, Gao F, Li H, Zhou S, Li H, et al. A framework for physiological indicators of flow in VR games: construction and preliminary evaluation. Pers Ubiquitous Comput. Springer London; 2016;20: 821–832. doi: 10.1007/s00779-016-0953-5

28. Alcañiz M, Baños R, Botella C, Rey B. The EMMA Project: Emotions as a Determinant of Presence. PsychNology J. 2003;1: 141–150.

29. Baños RM, Etchemendy E, Castilla D, García-palacios A, Quero S, Botella C. Positive mood induction procedures for virtual environments designed for elderly people. Interact Comput. 2012;24: 131–138. doi: 10.1016/j.intcom.2012.04.002

30. Riva G, Mantovani F, Capideville CS, Preziosa A, Morganti F, Villani D, et al. Affective Interactions Using Virtual Reality: The Link between Presence and Emotions. CyberPsychology Behav. 2007;10: 45–56. doi: 10.1089/cpb.2006.9993 17305448

31. Vecchiato G, Jelic A, Tieri G, Maglione AG, De Matteis F, Babiloni F. Neurophysiological correlates of embodiment and motivational factors during the perception of virtual architectural environments. Cogn Process. Springer Berlin Heidelberg; 2015;16: 425–429. doi: 10.1007/s10339-015-0725-6 26224275

32. Slater M, Wilbur S. A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence Teleoperators Virtual Environ. 1997;6: 603–616. doi: 10.1007/s10750-008-9541-7

33. Higuera-Trujillo JL, López-Tarruella J, Llinares MC. Psychological and physiological human responses to simulated and real environments: A comparison between Photographs, 360° Panoramas, and Virtual Reality. Appl Ergon. 2017;In Press. doi: 10.1016/j.apergo.2017.05.006 28601190

34. Bishop ID, Rohrmann B. Subjective responses to simulated and real environments: a comparison. Landsc Urban Plan. 2003;65: 261–277.

35. de Kort YAW, Ijsselsteijn WA, Kooijman J, Schuurmans Y. Virtual laboratories: comparability of real and virtual environments for environmental psychology. Presence Teleoperators Virtual Environ. 2003;12: 360–373.

36. Yeom D, Choi JH, Zhu Y. Investigation of physiological differences between immersive virtual environment and indoor environment in a building. Indoor Built Environ. 2017;0: 1–17.

37. van der Ham IJ, Faber AM, Venselaar M, van Kreveld MJ, Löffler M. Ecological validity of virtual environments to assess human navigation ability. Front Psychol. 2015;6: 637. doi: 10.3389/fpsyg.2015.00637 26074831

38. Villa C, Labayrade R. Validation of an online protocol for assessing the luminous environment. Light Res Technol. 2012;0: 1–20.

39. Eberhard JP. Applying Neuroscience to Architecture. Neuron. Elsevier Inc.; 2009;62: 753–756. doi: 10.1016/j.neuron.2009.06.001 19555644

40. Nanda U, Pati D, Ghamari H, Bajema R. Lessons from neuroscience: form follows function, emotions follow form. Intell Build Int. 2013;5: 61–78.

41. Russell JA. A circumplex model of affect. J Pers Soc Psychol. 1980;39: 1161–1178.

42. Sejima K. Kazuyo Sejima 1988–1996. El Croquis. 1996;15.

43. Slater M, Usoh M, Steed A. Depth of Presence in Virtual Environments. Presence Teleoperators Virtual Environ. 1994;3: 130–144. doi: 10.1162/pres.1994.3.2.130

44. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med. 2001;16: 606–613. doi: 10.1046/j.1525-1497.2001.016009606.x 11556941

45. Lang PJ, Bradley MM, Cuthbert BN. International Affective Picture System (IAPS): Technical Manual and Affective Ratings. NIMH Cent Study Emot Atten. 1997; 39–58. doi: 10.1027/0269-8803/a000147

46. Bradley MM, Lang PJ. Measuring emotion: The self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry. 1994;25: 49–59. doi: 10.1016/0005-7916(94)90063-9 7962581

47. Cousineau D, Chartier S. Outliers detection and treatment: a review. Int J Psy chological Res. 2010;3: 58–67. doi: 10.21500/20112084.844

48. Tarvainen MP, Ranta-aho PO, Karjalainen PA. An advanced detrending method with application to HRV analysis. IEEE Trans Biomed Eng. 2002;49: 172–175. doi: 10.1109/10.979357 12066885

49. Tarvainen MP, Niskanen JP, Lipponen JA, Ranta-aho PO, Karjalainen PA. Kubios HRV—Heart rate variability analysis software. Comput Methods Programs Biomed. Elsevier Ireland Ltd; 2014;113: 210–220. doi: 10.1016/j.cmpb.2013.07.024 24054542

50. Acharya UR, Joseph KP, Kannathal N, Lim CM, Suri JS. Heart rate variability: A review. Med Biol Eng Comput. 2006;44: 1031–1051. doi: 10.1007/s11517-006-0119-0 17111118

51. Richman J, Moorman J. Physiological time-series analysis using approximate entropy and sample entropy. Am J Physiol Hear Circ Physiol. 2000;278: H2039–H2049.

52. Pincus S, Viscarello R. Approximate Entropy A regularity measure for fetal heart rate analysis. Obstet Gynecol. 1992;79: 249–255. Available: file:///C:/Users/Admin/AppData/Local/Mendeley Ltd./Mendeley Desktop/Downloaded/Pincus, Viscarello—1992–1992—Approximate Entropy A regularity measure for fetal heart rate analysis.pdf.pdf 1731294

53. Peng C-K, Havlin S, Stanley HE, Goldberger AL. Quantification of scaling exponents and crossover phenomena in nonstationary heartbeat time series. Chaos. 1995;5: 82–87. doi: 10.1063/1.166141 11538314

54. Grassberger P, Procaccia I. Characterization of strange attractors. Phys Rev Lett. 1983;50: 346–349. doi: 10.1103/PhysRevLett.50.346

55. Delorme A, Makeig S. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods. 2004;134: 9–21. doi: 10.1016/j.jneumeth.2003.10.009 15102499

56. Colomer Granero A, Fuentes-Hurtado F, Naranjo Ornedo V, Guixeres Provinciale J, Ausín JM, Alcañiz Raya M. A Comparison of Physiological Signal Analysis Techniques and Classifiers for Automatic Emotional Evaluation of Audiovisual Contents. Front Comput Neurosci. 2016;10: 1–14. doi: 10.3389/fncom.2016.00001

57. Kober SE, Kurzmann J, Neuper C. Cortical correlate of spatial presence in 2D and 3D interactive virtual reality: An EEG study. Int J Psychophysiol. Elsevier B.V.; 2012;83: 365–374. doi: 10.1016/j.ijpsycho.2011.12.003 22206906

58. Hyvärinen A, Oja E. Independent component analysis: Algorithms and applications. Neural Networks. 2000;13: 411–430. doi: 10.1016/S0893-6080(00)00026-5 10946390

59. Mormann F, Lehnertz K, David P, E. Elger C. Mean phase coherence as a measure for phase synchronization and its application to the EEG of epilepsy patients. Phys D Nonlinear Phenom. 2000;144: 358–369. doi: 10.1016/S0167-2789(00)00087-7

60. Jolliffe IT. Principal Component Analysis, Second Edition. Encycl Stat Behav Sci. 2002;30: 487. doi: 10.2307/1270093

61. Schöllkopf B, Smola AJ, Williamson RC, Bartlett PL. New support vector algorithms. Neural Comput. 2000;12: 1207–1245. doi: 10.1162/089976600300015565 10905814

62. Yan K, Zhang D. Feature selection and analysis on correlated gas sensor data with recursive feature elimination. Sensors Actuators, B Chem. Elsevier B.V.; 2015;212: 353–363. doi: 10.1016/j.snb.2015.02.025

63. Chang C-C, Lin C-J. Libsvm: A Library for Support Vector Machines. ACM Trans Intell Syst Technol. 2011;2: 1–27. doi: 10.1145/1961189.1961199

64. Howell DC. Statistical methods for psychology. Cengage Learning; 2009.

65. Gorini A, Capideville CS, De Leo G, Mantovani F, Riva G. The Role of Immersion and Narrative in Mediated Presence: The Virtual Hospital Experience. Cyberpsychology, Behav Soc Netw. 2011;14: 99–105. doi: 10.1089/cyber.2010.0100 20649451

66. Glass L. Synchronization and rhythmic processes in physiology. Nature. 2001;410: 277–284. doi: 10.1038/35065745 11258383

67. Stam CJ. Nonlinear dynamical analysis of EEG and MEG: Review of an emerging field. Clin Neurophysiol. 2005;116: 2266–2301. doi: 10.1016/j.clinph.2005.06.011 16115797

68. Zhao Q Bin Zhang LQ, Cichocki A. EEG-based asynchronous BCI control of a car in 3D virtual reality environments. Chinese Sci Bull. 2009;54: 78–87. doi: 10.1007/s11434-008-0547-3

69. Baumgartner T, Valko L, Esslen M, Jäncke L. Neural Correlate of Spatial Presence in an Arousing and Noninteractive Virtual Reality: An EEG and Psychophysiology Study. CyberPsychology Behav. 2006;9: 30–45. doi: 10.1089/cpb.2006.9.30 16497116

70. Koelstra S, Mühl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, et al. DEAP: A dataset for emotion analysis using physiological and audiovisual signals. IEEE Trans Affect Comput. 2012;3: 1–15. Available: http://www.eecs.qmul.ac.uk/mmv/datasets/deap/

71. Kim J, André E. Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell. 2008;30: 2067–2083. doi: 10.1109/TPAMI.2008.26 18988943

72. Lin YP, Wang CH, Jung TP, Wu TL, Jeng SK, Duann JR, et al. EEG-based emotion recognition in music listening. IEEE Trans Biomed Eng. 2010;57: 1798–1806. doi: 10.1109/TBME.2010.2048568 20442037

73. Combrisson E, Jerbi K. Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. J Neurosci Methods. Elsevier B.V.; 2015;250: 126–136. doi: 10.1016/j.jneumeth.2015.01.010 25596422

74. de Borst AW, de Gelder B. Is it the real deal? Perception of virtual characters versus humans: An affective cognitive neuroscience perspective. Front Psychol. 2015;6: 1–12. doi: 10.3389/fpsyg.2015.00001

75. Mitchell RLC, Phillips LH. The overlapping relationship between emotion perception and theory of mind. Neuropsychologia. Elsevier; 2015;70: 1–10. doi: 10.1016/j.neuropsychologia.2015.02.018 25687032

76. Powers MB, Emmelkamp PMG. Virtual reality exposure therapy for anxiety disorders: A meta-analysis. J Anxiety Disord. 2008;22: 561–569. doi: 10.1016/j.janxdis.2007.04.006 17544252

77. Stein JP, Ohler P. Venturing into the uncanny valley of mind—The influence of mind attribution on the acceptance of human-like characters in a virtual reality setting. Cognition. Elsevier B.V.; 2017;160: 43–50. doi: 10.1016/j.cognition.2016.12.010 28043026

78. Critchley HD. Psychophysiology of neural, cognitive and affective integration: fMRI and autonomic indicants. Int J Psychophysiol. Elsevier B.V.; 2009;73: 88–94. doi: 10.1016/j.ijpsycho.2009.01.012 19414044

79. Niedenthal PM. Embodying emotion. Science (80-). 2007;316: 1002–1005. doi: 10.1126/science.1136930 17510358

80. Leer A, Engelhard IM, Van Den Hout MA. How eye movements in EMDR work: Changes in memory vividness and emotionality. J Behav Ther Exp Psychiatry. Elsevier Ltd; 2014;45: 396–401. doi: 10.1016/j.jbtep.2014.04.004 24814304

81. Gentili C. Why do we keep failing in identifying reliable biological markers in depression? J Evidence-Based Psychother. 2017;17: 69–84. doi: 10.24193/jebp.2017.2.4

82. Jo Y, Lee H, Cho A, Whang M. Emotion Recognition Through Cardiovascular Response in Daily Life Using KNN Classifier. Adv Comput Sci Ubiquitous Comput. 2017; 1451–1456.

83. Hassib M, Schneegass S, Eiglsperger P, Henze N, Schmidt A, Alt F. EngageMeter: A System for Implicit Audience Engagement Sensing Using Electroencephalography. Proc 2017 CHI Conf Hum Factors Comput Syst. 2017; 5114–5119. doi: 10.1145/3025453.3025669

84. Zhang S, Zhang Y, Sun Y, Member I, Thakor N. Graph theoretical analysis of EEG functional network during multi-workload flight simulation experiment in virtual reality environment. 2017; 3957–3960. doi: 10.1109/EMBC.2017.8037722 29060763

85. Debener S, Minow F, Emkes R, Gandras K, de Vos M. How about taking a low-cost, small, and wireless EEG for a walk? Psychophysiology. 2012;49: 1617–1621. doi: 10.1111/j.1469-8986.2012.01471.x 23013047

86. Marín-Morales J, Higuera-Trujillo JL, Juan C de, Llinares C, Guixeres J, Iñarra S, et al. Navigation comparison between a real and a virtual museum: time-dependent differences using a head mounted display. Interact Comput. 2019; doi: 10.1093/iwc/iwz018

Real vs. immersive-virtual emotional experience: Analysis of psycho-physiological patterns in a free exploration of an art museum

Souhrn

Klíčová slova:

Zdroje

PLOS One

INSIGHTS from European Respiratory Congress

Současné pohledy na riziko v parodontologii

Svět praktické medicíny 3/2024 (znalostní test z časopisu)

Kardiologické projevy hypereozinofilií

Střevní příprava před kolonoskopií