Skip to content
2000
Volume 18, Issue 7
  • ISSN: 2352-0965
  • E-ISSN: 2352-0973

Abstract

Emotion recognition, both verbal and non-verbal, is a crucial component of artificial intelligence, psychology, and human-computer interaction. Emotion recognition is an integral component that significantly contributes to the improvement of communication and interaction. The research endeavors to conduct a thorough analysis and synthesis of the most recent developments in Deep Learning (DL) and Machine Learning (ML) techniques. Specifically, the study concentrates on the recognition of both verbal and non-verbal emotions. In contrast to previous research concentrated on verbal or non-verbal emotion detection separately, the study attempts to reconcile the gap between the two by demonstrating how ML and DL can be utilized effectively to detect emotions. The study also examines new methods, including multimodal data and integration of contextual information. Additionally, the research has examined the ethical implications and difficulties associated with emotion detection technologies. Findings have also revealed the wide-reaching implications for various sectors, including healthcare, education, customer service, and entertainment, where comprehending human emotions plays a crucial role in enhancing user experience and outcomes. In conclusion, the study provides invaluable knowledge to practitioners and researchers, which may facilitate the development of more advanced and accurate systems.

© 2025 Bentham Science Publishers
Loading

Article metrics loading...

/content/journals/raeeng/10.2174/0123520965303142240430101645
2024-05-15
2025-09-26

  • From This Site

    /content/journals/raeeng/10.2174/0123520965303142240430101645
    dcterms_title,dcterms_subject,pub_keyword
    -contentType:Contributor -contentType:Concept -contentType:Institution
    10
    5
Loading full text...

Full text loading...

References

  1. EscobedoL. TentoriM. QuintanaE. FavelaJ. Garcia-RosasD. Using augmented reality to help children with autism stay focused.IEEE Pervasive Comput.2014131384610.1109/MPRV.2014.19
     [Google Scholar]
  2. SinghA. RajK. KumarT. VermaS. RoyA. Deep learning-based cost-effective and responsive robot for autism treatment.Drones2023728110.3390/drones7020081
     [Google Scholar]
  3. BrignellA. ChenauskyK.V. SongH. ZhuJ. SuoC. MorganA.T. Communication interventions for autism spectrum disorder in minimally verbal children.Cochrane Libr.2018201811CD01232410.1002/14651858.CD012324.pub230395694
     [Google Scholar]
  4. SarkerI.H. Deep learning: A comprehensive overview on techniques, taxonomy, applications and research directions.SN Computer Science20212642010.1007/s42979‑021‑00815‑134426802
     [Google Scholar]
  5. Mustaqeem KwonS. A CNN-assisted enhanced audio signal processing.Sensors201920118310.3390/s2001018331905692
     [Google Scholar]
  6. MandyW. WangA. LeeI. SkuseD. Evaluating social (pragmatic) communication disorder.J. Child Psychol. Psychiat.201758101166117510.1111/jcpp.1278528741680
     [Google Scholar]
  7. SinghS.K. ThakurR.K. KumarS. AnandR. Deep Learning and Machine Learning based Facial Emotion Detection using CNN.Proceedings of the 2022 9th International Conference on Computing for Sustainable Global Development, INDIACom 202223-25 March 2022, New Delhi, India, pp. 530-535, 2022.10.23919/INDIACom54597.2022.9763165
     [Google Scholar]
  8. KhullarV. TiwariR.G. AgarwalA.K. DuttaS. Physiological signals based anxiety detection using ensemble machine learning.Lect. Not. Net. Syst.202229159760810.1007/978‑981‑16‑4284‑5_53
     [Google Scholar]
  9. CoroG. BardelliS. CuttanoA. FossatiN. Automatic detection of potentially ineffective verbal communication for training through simulation in neonatology.Educ. Inf. Technol.20222779181920310.1007/s10639‑022‑11000‑z
     [Google Scholar]
  10. KhanA.R. Facial emotion recognition using conventional machine learning and deep learning methods: current achievements, analysis and remaining challenges.Information202213626810.3390/info13060268
     [Google Scholar]
  11. AlvesS. MarquesA. QueirósC. OrvalhoV. LifeisGame prototype: A serious game about emotions for children with autism spectrum disorders.PsychNology J.201311191211
     [Google Scholar]
  12. CeperoÁ. ClapésA. EscaleraS. Automatic non-verbal communication skills analysis: A quantitative evaluation.AI Commun.20152818710110.3233/AIC‑140617
     [Google Scholar]
  13. TeyeMT MissahYM AheneE FrimpongT Evaluation of Conversational Agents: Understanding Culture, Context and Environment in Emotion Detection.IEEE Access vol. 10, pp. 24976-24984, 2022.10.1109/ACCESS.2022.3153787
     [Google Scholar]
  14. RomeroN.L. A pilot study examining a computer-based intervention to improve recognition and understanding of emotions in young children with communication and social deficits.Res. Dev. Disabil.201765354510.1016/j.ridd.2017.04.00728433792
     [Google Scholar]
  15. MubinS.A. ThiruchelvamV. AndrewY.W. Extended Reality: How They Incorporated for ASD Intervention.2020 8th International Conference on Information Technology and Multimedia (ICIMU) Selangor, Malaysia, pp. 262-266, 2020.10.1109/ICIMU49871.2020.9243332
     [Google Scholar]
  16. HorstmannA.C. MühlL. KöppenL. LindhausM. StorchD. BührenM. RöttgersH.R. KrajewskiJ. Important preliminary insights for designing successful communication between a robotic learning assistant and children with autism spectrum disorder in germany.Robotics202211614110.3390/robotics11060141
     [Google Scholar]
  17. O’ByrneW.I. HouserK. StoneR. WhiteM. Digital storytelling in early childhood: Student illustrations shaping social interactions.Front. Psychol.20189180010.3389/fpsyg.2018.0180030364158
     [Google Scholar]
  18. GhoshT. BannaM.H.A. RahmanM.S. KaiserM.S. MahmudM. HosenA.S.M.S. ChoG.H. Artificial intelligence and internet of things in screening and management of autism spectrum disorder.Sustain Cities Soc.20217410318910.1016/j.scs.2021.103189
     [Google Scholar]
  19. D’AlfonsoS. AI in mental health.Curr. Opin. Psychol.20203611211710.1016/j.copsyc.2020.04.00532604065
     [Google Scholar]
  20. TrigeorgisG. RingevalF. BruecknerR. MarchiE. NicolaouM.A. SchullerB. Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network.2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 20-25 March 2016, Shanghai, China, pp. 5200-5204, 2016.10.1109/ICASSP.2016.7472669
     [Google Scholar]
  21. Al BannaM.H. GhoshT. TaherK.A. KaiserM.S. MahmudM. A monitoring system for patients of autism spectrum disorder using artificial intelligence.Lect. Not. Comput. Sci.20201224125126210.1007/978‑3‑030‑59277‑6_23
     [Google Scholar]
  22. PoriaS. CambriaE. HussainA. HuangG.B. Towards an intelligent framework for multimodal affective data analysis.Neural Netw.20156310411610.1016/j.neunet.2014.10.00525523041
     [Google Scholar]
  23. TayeM.M. Understanding of machine learning with deep learning: architectures, workflow, applications and future directions.Computers20231259110.3390/computers12050091
     [Google Scholar]
  24. AbbaschianB.J. Sierra-SosaD. ElmaghrabyA. Deep learning techniques for speech emotion recognition, from databases to models.Sensors2021214124910.3390/s2104124933578714
     [Google Scholar]
  25. YuanQ. A classroom emotion recognition model based on a convolutional neural network speech emotion algorithm.Occup. Ther. Int.2022202211210.1155/2022/956387735912313
     [Google Scholar]
  26. NasimA.S. ChowdoryR.H. DeyA. DasA. Recognizing Speech Emotion Based on Acoustic Features Using Machine Learning.2021 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2021 23-25 October 2021, Depok, Indonesia, pp. 1-7, 2021.10.1109/ICACSIS53237.2021.9631319
     [Google Scholar]
  27. YangB. HanX. TangJ. Three Class Emotions Recognition Based On Deep Learning Using Staked Autoencoder.2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI 2017)14-16 October 2017, Shanghai, China, p. 1–5, 2017.
     [Google Scholar]
  28. Marin VargasA. CominelliL. Dell’OrlettaF. ScilingoE.P. Verbal communication in robotics: a study on salient terms, research fields and trends in the last decades based on a computational linguistic analysis.Front. Comput. Sci.2021259116410.3389/fcomp.2020.591164
     [Google Scholar]
  29. ScahillL. JekelJ.F. SchillingL.S. Screening child psychiatric inpatients for communication disorders: A pilot study.Arch. Psychiatr. Nurs.199151313710.1016/0883‑9417(91)90007‑R2039279
     [Google Scholar]
  30. HossainM.S. MuhammadG. Emotion recognition using deep learning approach from audio–visual emotional big data.Inf. Fusion201949697810.1016/j.inffus.2018.09.008
     [Google Scholar]
  31. MoghadasM. MoradiH. Analyzing human-robot interaction using machine vision for autism screening.2018 6th RSI International Conference on Robotics and Mechatronics (IcRoM)23-25 October 2018, Tehran, Iran, pp. 572-576, 2018.10.1109/ICRoM.2018.8657569
     [Google Scholar]
  32. TrR. LilhoreU.K. MP. SimaiyaS. KaurA. HamdiM. Predictive analysis of heart diseases with machine learning approaches.Malays. J. Comput. Sci.202213214810.22452/mjcs.sp2022no1.10
     [Google Scholar]
  33. GuleriaK. SharmaS. KumarS. TiwariS. Early prediction of hypothyroidism and multiclass classification using predictive machine learning and deep learning.Measurement: Sensors20222410048210.1016/j.measen.2022.100482
     [Google Scholar]
  34. KhullarV. SinghH. BalaM. Deep neural network-based handheld diagnosis system for Autism spectrum disorder.Neurol. India2021691667410.4103/0028‑3886.31006933642273
     [Google Scholar]
  35. ChoudharyR.R. MeenaG. MohbeyK.K. Speech emotion based sentiment recognition using deep neural networks.J. Phys. Conf. Ser.20222236101200310.1088/1742‑6596/2236/1/012003
     [Google Scholar]
  36. YoonS. ByunS. JungK. Multimodal Speech Emotion Recognition Using Audio and Text.2018 IEEE Spoken Language Technology Workshop (SLT)18-21 December 2018, Athens, Greece, pp. 112-118, 2018.10.1109/SLT.2018.8639583
     [Google Scholar]
  37. FarhatN.H. Photonit neural networks and learning mathines the role of electron-trapping materials.IEEE Expert-Intelligent Systems and Their Applications19927637210.1109/64.163674
     [Google Scholar]
  38. LaackeS. MuellerR. SchomerusG. SallochS. Artificial intelligence, social media and depression. a new concept of health-related digital autonomy.Am. J. Bioeth.202121742010.1080/15265161.2020.186351533393864
     [Google Scholar]
  39. IrelandD. BradfordD. Farr-WhartonG. Social fringe dwellers: Can chat-bots combat bullies to improve participation for children with autism?Journal of Community Informatics201814110511910.15353/joci.v14i1.3405
     [Google Scholar]
  40. SubramanianR.R. SireeshaY. ReddyY.S.P.K. BindamruthaT. HarikaM. SudharsanR.R. Audio Emotion Recognition by Deep Neural Networks and Machine Learning Algorithms.2021 International Conference on Advancements in Electrical, Electronics, Communication, Computing and Automation, ICAECA 202108-09 October 2021, Coimbatore, India, pp. 1-6, 2021.10.1109/ICAECA52838.2021.9675492
     [Google Scholar]
  41. LucarelliG. BorrottiM. A Hybrid Model Based on Fuzzy Rules to Act on the Diagnosed of Autism in Adults.15th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI)May 2019, Hersonissos, Greece. Vol. 2, pp. 401-412, 2019.10.1007/978‑3‑030‑19823‑7
     [Google Scholar]
  42. MenegottoA.B. BeckerC.D.L. CazellaS.C. CazellaS.C. Computer-aided diagnosis of hepatocellular carcinoma fusing imaging and structured health data.Health Inf. Sci. Syst.2021912010.1007/s13755‑021‑00151‑x33968399
     [Google Scholar]
  43. PanJ FangW ZhangZ ChenB ZhangZ WangS. Multimodal Emotion Recognition based on Facial Expressions, Speech, and EEG.IEEE Open Journal of Engineering in Medicine and Biologypp. 1-6, 2023.10.1109/OJEMB.2023.3240280
     [Google Scholar]
  44. AlsharekhM.F. Facial emotion recognition in verbal communication based on deep learning.Sensors20222216610510.3390/s2216610536015866
     [Google Scholar]
  45. Villegas-ChWE Garcia-OrtizJ Sanchez-ViteriS Identification of emotions from facial gestures in a teaching environment with the use of machine learning techniques.IEEE Access202311380103802210.1109/ACCESS.2023.3267007
     [Google Scholar]
  46. SchoneveldL. OthmaniA. AbdelkawyH. Leveraging recent advances in deep learning for audio-Visual emotion recognition.Pattern Recognit. Lett.20211461710.1016/j.patrec.2021.03.007
     [Google Scholar]
  47. DantasA.C. do NascimentoM.Z. Recognition of emotions for people with autism: An approach to improve skills.Int. J. Comput. Games Technol.2022202212110.1155/2022/6738068
     [Google Scholar]
  48. PuriT. SoniM. DhimanG. Ibrahim KhalafO. AlazzamM. Raza KhanI. Detection of emotion of speech for RAVDESS audio using hybrid convolution neural network.J. Healthc. Eng.2022202210.1155/2022/847294735265307
     [Google Scholar]
  49. TrivediN.K. GautamV. AnandA. AljahdaliH.M. VillarS.G. AnandD. GoyalN. KadryS. Early detection and classification of tomato leaf disease using high-performance deep neural network.Sensors20212123798710.3390/s2123798734883991
     [Google Scholar]
  50. IfteneA. TrandabățD. Enhancing the attractiveness of learning through augmented reality.Procedia Comput. Sci.201812616617510.1016/j.procs.2018.07.220
     [Google Scholar]
  51. PalestraG. De CarolisB. EspositoF. Artificial intelligence for robot-assisted treatment of autism.CEUR Workshop Proc.201719821724
     [Google Scholar]
  52. XunL.U. An argument on basic emotions.Cogn. Emotion2022798010.2307/j.ctv2vr9cj4.23
     [Google Scholar]
  53. GabbatoreI. MarchettiA. BoscoFM. The fuzzy boundaries of the social (pragmatic) communication disorder (SPCD): Why the picture is still so confusing?Heliyon202398e1906210.1016/j.heliyon.2023.e1906237664706PMC10468801
     [Google Scholar]
  54. ParvathiD.S.L. LeelavathiN. RavikumarJ.M.S.V. SujathaB. Emotion analysis using deep learning.Proceedings of the International Conference on Electronics and Sustainable Communication Systems (ICESC 2020)02-04 July 2020, Coimbatore, India, pp. 593-598, 2020.10.1109/ICESC48915.2020.9155948
     [Google Scholar]
  55. SidlauskieneJ. JoyeY. AuruskevicieneV. AI-based chatbots in conversational commerce and their effects on product and price perceptions.Electron. Mark.20233312410.1007/s12525‑023‑00633‑837252674
     [Google Scholar]
  56. ZhangJ. YinZ. ChenP. NicheleS. Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review.Inf. Fusion20205910312610.1016/j.inffus.2020.01.011
     [Google Scholar]
  57. HizlisoyS YildirimS TufekciZ Music emotion recognition using convolutional long short term memory deep neural networks.Enginee. Sci. Technol. Int. J.202124376076710.1016/j.jestch.2020.10.009
     [Google Scholar]
  58. AhsanMM LunaSA SiddiqueZ Machine-learning-based disease diagnosis: A comprehensive review.Healthcare202210354110.3390/healthcare1003054135327018PMC8950225
     [Google Scholar]
  59. PoriaS. HazarikaD. MajumderN. NaikG. CambriaE. MihalceaR. MELD: A multimodal multi-party dataset for emotion recognition in conversations.Proceedings of the 57th Annual Meeting of the Association for Computational LinguisticsJuly 2019, Florence, Italy, pp. 527-536, 2020.10.18653/v1/P19‑1050
     [Google Scholar]
  60. NagaP. MarriS.D. BorreoR. Facial emotion recognition methods, datasets and technologies: A literature survey.Mater. Today Proc.2023802824282810.1016/j.matpr.2021.07.046
     [Google Scholar]
  61. ChungC.H. ChenC.H. Augmented reality based social stories training system for promoting the social skills of children with autism.Adv. Intell. Syst. Comput.201748649550510.1007/978‑3‑319‑41685‑4_44
     [Google Scholar]
  62. ChawlaM. Technological Intervention for Supporting Individuals for Social (Pragmatic) Communication DisordersInternational Conference on Reliability, Infocom Technologies and Future Directions (ICRITO)13-14 October 2022, Noida, India, pp. 1-5, 2022.10.1109/ICRITO56286.2022.9965160
     [Google Scholar]
  63. AdamsC. LocktonE. CollinsA. Metapragmatic explicitation and social attribution in social communication disorder and developmental language disorder: A comparative study.J. Speech Lang. Hear. Res.201861360461810.1044/2017_JSLHR‑L‑17‑002629471463
     [Google Scholar]
  64. MeddageP. EkanayakeI. PereraU.S. AzamathullaH.M. Md SaidM.A. RathnayakeU. Interpretation of machine-learning-based (black-box) wind pressure predictions for low-rise gable-roofed buildings using shapley additive explanations (SHAP).Buildings202212673410.3390/buildings12060734
     [Google Scholar]
  65. EkanayakeI.U. MeddageD.P.P. RathnayakeU. A novel approach to explain the black-box nature of machine learning in compressive strength predictions of concrete using Shapley additive explanations (SHAP).Cas. Stud. Construc. Mater.202216e0105910.1016/j.cscm.2022.e01059
     [Google Scholar]
  66. LinardatosP. PapastefanopoulosV. KotsiantisS. Explainable ai: A review of machine learning interpretability methods.Entropy20202311810.3390/e2301001833375658
     [Google Scholar]
  67. AS. RS. A systematic review of Explainable Artificial Intelligence models and applications: Recent developments and future trends.Decis. Analyt. J.2023710023010.1016/j.dajour.2023.100230
     [Google Scholar]
  68. MeddageD.P.P. EkanayakeI.U. WeerasuriyaA.U. LewangamageC.S. TseK.T. MiyanawalaT.P. RamanayakaC.D.E. Explainable Machine Learning (XML) to predict external wind pressure of a low-rise building in urban-like settings.J. Wind Eng. Ind. Aerodyn.202222610502710.1016/j.jweia.2022.105027
     [Google Scholar]
  69. KimS.H. LordC. Combining information from multiple sources for the diagnosis of autism spectrum disorders for toddlers and young preschoolers from 12 to 47 months of age.J. Child Psychol. Psychiat.201253214315110.1111/j.1469‑7610.2011.02458.x21883205
     [Google Scholar]
  70. MalinverniL. Mora-GuiardJ. PadilloV. ValeroL. HervásA. ParesN. An inclusive design approach for developing video games for children with Autism Spectrum Disorder.Comput. Human Behav.20177153554910.1016/j.chb.2016.01.018
     [Google Scholar]
  71. AslamAR HafeezN HeidariH Channels and features identification: A review and a machine-learning based model with large scale feature extraction for emotions and asd classificationFront Neurosci20221684485110.3389/fnins.2022.84485135937896PMC9355483
     [Google Scholar]
  72. LuX. Deep learning based emotion recognition and visualization of figural representation deep learning and emotion recognition.Front. Psychol.20221281883310.3389/fpsyg.2021.81883335069400
     [Google Scholar]
  73. NaidooS.W. NaickerN. PatelS.S. GovenderP. Computer vision: The effectiveness of deep learning for emotion detection in marketing campaigns.Int. J. Adv. Comput. Sci. Appl.202213588389010.14569/IJACSA.2022.01305100
     [Google Scholar]
  74. AnX.D. RuanZ. Speech emotion recognition algorithm based on deep learning algorithm fusion of temporal and spatial features.J. Phys. Conf. Ser.20211861101206410.1088/1742‑6596/1861/1/012064
     [Google Scholar]
  75. DingW XuM HuangD LinW DongM YuX Audio and Face Video Emotion Recognition in the Wild using Deep Neural Networks and Small DatasetsIn Proceedings of the 18th ACM International Conference on Multimodal Interaction (ICMI '16). Association for Computing MachineryNew York, NY, USA, pp. 506–513, 2016.10.1145/2993148.2997637
     [Google Scholar]
  76. DuanM. LiK. OuyangA. WinK.N. LiK. TianQ. EGroupNet: A feature-enhanced network for age estimation with novel age group schemesACM Trans. Multimed. Comput. Commun. Appl.202016212310.1145/3379449
     [Google Scholar]
  77. GeorgescuA.L. KoehlerJ.C. WeiskeJ. VogeleyK. KoutsoulerisN. Falter-WagnerC. Machine learning to study social interaction difficulties in ASD.Front. Robot. AI2019613210.3389/frobt.2019.0013233501147
     [Google Scholar]
  78. LéveilléC. BarbeauE.B. BolducC. LimogesÉ. BerthiaumeC. ChevrierÉ. MottronL. GodboutR. Enhanced connectivity between visual cortex and other regions of the brain in autism: a REM sleep EEG coherence study.Autism Res.20103528028510.1002/aur.15520717953
     [Google Scholar]
  79. PomazanV. TvoroshenkoI. GorokhovatskyiV. Development of an Application for Recognizing Emotions Using Convolutional Neural Networks.2021 IEEE 7th International Conference on Smart Instrumentation, Measurement and Applications (ICSIMA)23-25 August 2021, Bandung, Indonesia, pp. 47-52, 2021.
     [Google Scholar]
  80. BonoV. NarzisiA. JouenA.L. TilmontE. HommelS. JamalW. XavierJ. BilleciL. MaharatnaK. WaldM. ChetouaniM. CohenD. MuratoriF. MICHELANGELO Study Group GOLIAH: A gaming platform for home-based intervention in autism - principles and design.Front. Psychiatry201677010.3389/fpsyt.2016.0007027199777
     [Google Scholar]
  81. TalegaonkarI. JoshiK. ValunjS. KohokR. KulkarniA. Real time facial expression recognition using deep learning.Proceedings of International Conference on Communication and Information Processing (ICCIP)Karvenagar, Pune, 30 Sep 2019, pp. 1-8, 2019.10.2139/ssrn.3421486
     [Google Scholar]
  82. SinghalB.S. KhadilkarS.V. Neurology in the developing world.Handb Clin Neurol20141211773178210.1016/B978‑0‑7020‑4088‑7.00114‑0
     [Google Scholar]
  83. KumarS. PrabhaR. SamuelS. Sentiment analysis and emotion detection with healthcare perspective.Stud. Computat. Intell.2022102418920410.1007/978‑981‑19‑1076‑0_11
     [Google Scholar]
  84. MadhusankaB.G.D.A. RamadassS. Implicit intention communication for activities of daily living of elder/disabled people to improve well-being.Stud. Computat. Intell.202193332534210.1007/978‑981‑15‑9897‑5_16
     [Google Scholar]
  85. Pichora-FullerM.K. DupuisK. Toronto emotional speech set (TESS).Schol. Port. Datav.20201202010.5683/SP2/E8H2MF
     [Google Scholar]
  86. KhullarV. SinghH.P. BalaM. Meltdown/Tantrum Detection System(MTDS) for individuals with autism spectrum disorder.Appl. Artif. Intell.202135151708173210.1080/08839514.2021.1991115
     [Google Scholar]
  87. LiH. SunJ. XuZ. ChenL. Multimodal 2D+3D facial expression recognition with deep fusion convolutional neural network.IEEE Trans. Multimed.201719122816283110.1109/TMM.2017.2713408
     [Google Scholar]
  88. GautamV. TrivediN.K. SinghA. MohamedH.G. NoyaI.D. KaurP. GoyalN. A transfer learning-based artificial intelligence model for leaf disease assessment.Sustainability202214201361010.3390/su142013610
     [Google Scholar]
  89. UzuegbunamN. WongW.H. CheungS.C.S. RubleL. MEBook: Multimedia social greetings intervention for children with autism spectrum disorders.IEEE Trans. Learn. Technol.201811452053510.1109/TLT.2017.2772255
     [Google Scholar]
  90. Yongjin Wang Ling Guan VenetsanopoulosA.N. Kernel cross-modal factor analysis for information fusion with application to bimodal emotion recognition.IEEE Trans. Multimed.201214359760710.1109/TMM.2012.2189550
     [Google Scholar]
  91. HaoxiangS. ChangxingC. YunfeiL. MuY. Cooperative perception optimization based on self - checking machine learning.Comput. Mater. Continua202062274776110.32604/cmc.2020.05625
     [Google Scholar]
  92. YangH. YinL. CNN based 3D facial expression recognition using masking and landmark features2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)23-26 October 2017, San Antonio, TX, USA, pp. 556-560, 2017.10.1109/ACII.2017.8273654
     [Google Scholar]
  93. GlowinskiD. CamurriA. VolpeG. DaelN. SchererK. Technique for automatic emotion recognition by body gesture analysis2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops23-28 June 2008, Anchorage, AK, USA, pp. 1-6, 2008.10.1109/CVPRW.2008.4563173
     [Google Scholar]
  94. BussoC. BulutM. LeeC.C. KazemzadehA. MowerE. KimS. ChangJ.N. LeeS. NarayananS.S. IEMOCAP: interactive emotional dyadic motion capture database.Lang. Resour. Eval.200842433535910.1007/s10579‑008‑9076‑6
     [Google Scholar]
  95. VallesD. MatinR. An Audio Processing Approach using Ensemble Learning for Speech-Emotion Recognition for Children with ASD2021 IEEE World AI IoT Congress (AIIoT)10-13 May 2021, Seattle, WA, USA, pp. 55-61, 2021.10.1109/AIIoT52608.2021.9454174
     [Google Scholar]
  96. HossainM.S. MuhammadG. Audio-visual emotion recognition using multi-directional regression and Ridgelet transform.J. Multimod. User Interf.201610432533310.1007/s12193‑015‑0207‑2
     [Google Scholar]
  97. EggerM. LeyM. HankeS. Emotion recognition from physiological signal analysis: A review.Electron. Notes Theor. Comput. Sci.2019343355510.1016/j.entcs.2019.04.009
     [Google Scholar]
  98. FranceseR. GuercioA. RossanoV. BhatiD. A Multimodal Conversational Interface to Support the creation of customized Social Stories for People with ASD.ACM International Conference Proceeding Series06 June 2022, New York, NY, USA, pp. 1-5, 2022.10.1145/3531073.3531118
     [Google Scholar]
  99. TzirakisP. TrigeorgisG. NicolaouM.A. SchullerB.W. ZafeiriouS. End-to-End multimodal emotion recognition using deep neural networks.IEEE J. Sel. Top. Signal Process.20171181301130910.1109/JSTSP.2017.2764438
     [Google Scholar]
  100. BergnerA.S. HildebrandC. HäublG. Machine talk: How verbal embodiment in conversational ai shapes consumer–brand relationships.J. Consum. Res.202350474276410.1093/jcr/ucad014
     [Google Scholar]
  101. LiX. ZhongH. ZhangB. ZhangJ. A general chinese chatbot based on deep learning and its’ application for children with ASD.Int. J. Mach. Learn. Comput.202010451952610.18178/ijmlc.2020.10.4.967
     [Google Scholar]
  102. IpH.H.S. WongS.W.L. ChanD.F.Y. ByrneJ. LiC. YuanV.S.N. LauK.S.Y. WongJ.Y.W. Enhance emotional and social adaptation skills for children with autism spectrum disorder: A virtual reality enabled approach.Comput. Educ.201811711510.1016/j.compedu.2017.09.010
     [Google Scholar]
  103. AndrewG. AroraR. BilmesJ. LivescuK. Deep Canonical Correlation AnalysisPMLR201328312471255
     [Google Scholar]
  104. LiuW. QiuJ.L. ZhengW.L. LuB.L. Comparing recognition performance and robustness of multimodal deep learning models for multimodal emotion recognition.IEEE Trans. Cogn. Dev. Syst.202214271572910.1109/TCDS.2021.3071170
     [Google Scholar]
  105. LeeI.J. ChenC.H. WangC.P. ChungC.H. Augmented reality plus concept map technique to teach children with ASD to use social cues when meeting and greeting.Asia-Pac. Educ. Res.201827322724310.1007/s40299‑018‑0382‑5
     [Google Scholar]
  106. WangY QiuS LiD DuC LuBL HeH Multi-modal domain adaptation variational autoencoder for EEG-based emotion recognition.IEEE/CAA J. Automat. Sini.202291612162610.1109/JAS.2022.105515
     [Google Scholar]
  107. LiY. WangY. YangX. ImS.K. Speech emotion recognition based on Graph-LSTM neural network.EURASIP J. Audio Speech Music Process.2023202314010.1186/s13636‑023‑00303‑9
     [Google Scholar]
  108. NandiA. XhafaF. A federated learning method for real-time emotion state classification from multi-modal streaming.Methods202220434034710.1016/j.ymeth.2022.03.00535314343
     [Google Scholar]
  109. LiL. FanY. TseM. LinK.Y. A review of applications in federated learning.Comput. Ind. Eng.202014910685410.1016/j.cie.2020.106854
     [Google Scholar]
  110. CheL. WangJ. ZhouY. MaF. Multimodal federated learning: A survey.Sensors20232315698610.3390/s2315698637571768
     [Google Scholar]
  111. GuptaC. KhullarV. GoyalN. SainiK. BaniwalR. KumarS. RastogiR. Cross-Silo, privacy-preserving, and lightweight federated multimodal system for the identification of major depressive disorder using audio and electroencephalogram.Diagnostics20231414310.3390/diagnostics1401004338201350
     [Google Scholar]
  112. RiekeN. HancoxJ. LiW. MilletarìF. RothH.R. AlbarqouniS. BakasS. GaltierM.N. LandmanB.A. Maier-HeinK. OurselinS. ShellerM. SummersR.M. TraskA. XuD. BaustM. CardosoM.J. The future of digital health with federated learning.NPJ Digit. Med.20203111910.1038/s41746‑020‑00323‑133015372
     [Google Scholar]
  113. ZhangT FengT AlamS LeeS ZhangM NarayananSS FedAudio: A Federated Learning Benchmark for Audio TasksICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)04-10 June 2023, Rhodes Island, Greece, pp. 1-5, 2023.10.1109/ICASSP49357.2023.10096500
     [Google Scholar]
  114. KhanT. NguyenK. MichalasA. A more secure split: Enhancing the security of privacy-preserving split learning.Lect. Notes Comput. Sci.20241432430732910.1007/978‑3‑031‑47748‑5_17
     [Google Scholar]
  115. AbediA KhanSS Federated split learning on distributed sequential data in recurrent neural networks.Multimed Tools Appl202311110.1007/s11042‑023‑15184‑5
     [Google Scholar]
  116. ChowdharyM AlgburiS. A collective review of Terahertz technology integrated with a newly proposed split learning based algorithm for healthcare systemUnivers. Bahrain Scienti. J.2010110
     [Google Scholar]
  117. MohammadH.M. AffectNet: A new database for facial expression, valence, and arousal computation in the wild.IEEE Transact. Affect. Comput.20172124
     [Google Scholar]
  118. NojavanasghariB. BaltrušaitisT. HughesC.E. MorencyL.P. Emo react: A multimodal approach and dataset for recognizing emotional responses in children.ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal InteractionOctober 2016, New York, NY, USA, pp. 137–144, 2016.10.1145/2993148.2993168
     [Google Scholar]
  119. OnF. MELD: Multimodal emotionlines dataset.Example dialogue n.d.20182224
     [Google Scholar]
  120. FarnsworthB. The international affective picture system.Imotions: Res. Fundement.2020111
     [Google Scholar]
  121. BrancoD. GonçalvesÓ.F. BadiaS.B. A systematic review of international affective picture system (IAPS) around the world.Sensors2023238386610.3390/s2308386637112214
     [Google Scholar]
  122. UtomoT.S. Emotion Label from ANEW dataset for Searching Best Definition from WordNet.2018 International Seminar on Application for Technology of Information and Communication21-22 September 2018, Semarang, Indonesia, pp. 249–252, 2018.10.1109/ISEMANTIC.2018.8549769
     [Google Scholar]
  123. LivingstoneS.R. RussoF.A. The ryerson audio-visual database of emotional speech and song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in north american englishPLoS One2018135e019639110.1371/journal.pone.019639129768426
     [Google Scholar]
  124. KoelstraS. MühlC. SoleymaniM. Jong-Seok Lee YazdaniA. EbrahimiT. PunT. NijholtA. PatrasI. DEAP: A database for emotion analysis; Using physiological signals.IEEE Trans. Affect. Comput.201231183110.1109/T‑AFFC.2011.15
     [Google Scholar]
  125. LewisB. TempletonG. LuoX. A scientometric investigation into the validity of IS journal quality measures.J. Assoc. Inf. Syst.200781261963310.17705/1jais.00145
     [Google Scholar]
  126. JupalleH. KouserS. BhatiaA.B. AlamN. NadikattuR.R. WhigP. Automation of human behaviors and its prediction using machine learning.Microsyst. Technol.20222881879188710.1007/s00542‑022‑05326‑4
     [Google Scholar]
  127. LordC. RisiS. LambrechtL. CookE.H.Jr LeventhalB.L. DiLavoreP.C. PicklesA. RutterM. The autism diagnostic observation schedule-generic: a standard measure of social and communication deficits associated with the spectrum of autism.J. Autism Dev. Disord.200030320522310.1023/A:100559240194711055457
     [Google Scholar]
  128. PujolF.A. MoraH. MartínezA. Emotion recognition to improve e-healthcare systems in smart cities. VisviziA. Lytrasand M. Research & Innovation ForumSpringer Proceedings in Complexity. SpringerCham201924525410.1007/978‑3‑030‑30809‑4_23
     [Google Scholar]
  129. WuW. AuL. JordanB. StathopoulosT. BatalinM. KaiserW. The SmartCane system: An assistive device for geriatrics.3rd International ICST Conference on Body Area NetworksMay 2010, California, Los Angeles, pp. 1-6, 2011.10.4108/ICST.BODYNETS2008.2944
     [Google Scholar]
  130. RibeiroB. OliveiraG. LaranjeiraA. ArraisJ.P. Deep learning in digital marketing: brand detection and emotion recognition.Int. J. Mach. Intell. Sens. Sig. Process.2017213210.1504/IJMISSP.2017.088173
     [Google Scholar]
  131. HussainS.S. PeterC. BieberG. Emotion recognition on the go: Providing personalized services based on emotional states.Comput. Sci. Psychol.200916
     [Google Scholar]
  132. KumazakiH. WarrenZ. MuramatsuT. YoshikawaY. MatsumotoY. MiyaoM. NakanoM. MizushimaS. WakitaY. IshiguroH. MimuraM. MinabeY. KikuchiM. A pilot study for robot appearance preferences among high-functioning individuals with autism spectrum disorder: Implications for therapeutic use.PLoS One20171210e018658110.1371/journal.pone.018658129028837
     [Google Scholar]
  133. AhireN.K. AwaleR.N. WaghA. Attention module-based fused deep cnn for learning disabilities identification using EEG signal.Multimed Tools ApplSpringer20231610.1007/s11042‑023‑17277‑7
     [Google Scholar]
  134. DykstraJ.R. BoydB.A. WatsonL.R. CraisE.R. BaranekG.T. The impact of the Advancing Social-communication And Play (ASAP) intervention on preschoolers with autism spectrum disorder.Autism2012161274410.1177/136236131140893321788256
     [Google Scholar]
  135. KleefGA Van K CôtéS The social effects of emotions.Annu Rev Psychol20227362965810.1146/annurev‑psych‑020821‑01085534280326
     [Google Scholar]
  136. KozimaH. YasudaY. NakagawaC. Social interaction facilitated by a minimally-designed robot: Findings from longitudinal therapeutic practices for autistic childrenRO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication26-29 August 2007, Jeju, Korea (South), pp. 599-604, 2007.10.1109/ROMAN.2007.4415155
     [Google Scholar]
  137. ChenW.H. HsuS.H. ShenH.P. Application of SVM and ANN for intrusion detection.Comput. Oper. Res.200532102617263410.1016/j.cor.2004.03.019
     [Google Scholar]
  138. XuD. YapanelU. GrayS. Reliability of the LENATM language environment analysis system in young children’s natural home environment.LENA Techn. Rep.20095116
     [Google Scholar]
  139. FengT. NarayananS. Semi-FedSER: Semi-supervised learning for speech emotion recognition on federated learning using multiview pseudo-labeling.Proc. Interspeech20225050505410.21437/Interspeech.2022‑141
     [Google Scholar]
  140. KatiraiA Ethical considerations in emotion recognition technologies: a review of the literatureAI EthicsSpringerLink202312310.1007/s43681‑023‑00307‑3
     [Google Scholar]
  141. KimY.S. HanD.S. Analysis of consultations by the Committee for Publication Ethics of the Korean Association of Medical Journal Editors.Sci. Ed.20207218418810.6087/kcse.215
     [Google Scholar]
  142. MusmadeP.B. NijhawanL.P. UdupaN. BairyK.L. BhatK.M. JanodiaM.D. MuddukrishnaB.S. Informed consent: Issues and challenges.J. Adv. Pharm. Technol. Res.20134313414010.4103/2231‑4040.11677924083200
     [Google Scholar]
  143. ReaderOI Ethical considerations in AI and data science: Bias, fairness, and accountability.Int. J. Infor. Tech. Manage. Info. Syst.20191011120
     [Google Scholar]
  144. Niral SutariaC.I.S.A. ACA. bias and ethical concerns in machine learning.Isaca2022413
     [Google Scholar]
  145. MoranS. Surveillance ethics.Philosophy Now20151101415
     [Google Scholar]
  146. Da BormidaM. The Big Data World: Benefits.Threats and Ethical Challenges2021719110.1108/S2398‑601820210000008007
     [Google Scholar]
  147. FluencyD LiteracyD. Navigate ethical challenges related to accountability and transparency when using ai applications.Data Fluen. Data Liter.202315
     [Google Scholar]
  148. PollH. Survey results: transparency and accountability key to ethical behaviour.The Account. onlin.202315
     [Google Scholar]
  149. Inclusivity in global research reporting guidelines for specific study types.PLoS One202312
     [Google Scholar]
  150. Van NiekerkT. A few critical observations of current oversight and accoun- tability challenges of financial management of municipalities in the Free State province: An ethical dilemma.J. Christian Scholarship20202020269293
     [Google Scholar]
  151. KesselsR.P.C. MontagneB. HendriksA.W. PerrettD.I. de HaanE.H.F. Assessment of perception of morphed facial expressions using the emotion recognition task: Normative data from healthy participants aged 8–75.J. Neuropsychol.201481759310.1111/jnp.1200923409767
     [Google Scholar]
  152. AloufiR HaddadiH BoyleD. Emotionless: Privacy-preserving speech analysis for voice assistantsCryptography and Security20191410.48550/arXiv.1908.03632
     [Google Scholar]
  153. StarkL. HoeyJ. The Ethics of Emotion in Artificial Intelligence Systems.FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and TransparencyMarch 2021, New York, NY, USA, pp. 782–793, 2021.10.1145/3442188.3445939
     [Google Scholar]
/content/journals/raeeng/10.2174/0123520965303142240430101645
Loading
Recent Trends in Machine and Deep Learning for Verbal and Non-verbal Emotion Detection
Recent Advances in Electrical & Electronic Engineering 18, 903 (2025); https://doi.org/10.2174/0123520965303142240430101645
/content/journals/raeeng/10.2174/0123520965303142240430101645
/content/journals/raeeng/10.2174/0123520965303142240430101645
Loading

Data & Media loading...


  • Article Type:     Review Article
Keyword(s): Artificial intelligence; assistive technologies; deep learning; emotion recognition; emotions; machine learning; nonverbal emotions; verbal emotions

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error
Please enter a valid_number test