Fri, Sep 27, 2024
| فارسی
Volume 26, Issue 1 (Spring 2024)
Advances in Cognitive Sciences 2024, 26(1): 91-108 |
Back to browse issues page
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Joodi N, Keshtiari N. A review of emotional prosody research based on functional magnetic resonance imaging and event-related potential techniques and an introduction to the Persian emotional speech database (Persian ESD). Advances in Cognitive Sciences 2024; 26 (1) :91-108
URL: http://icssjournal.ir/article-1-1634-en.html
URL: http://icssjournal.ir/article-1-1634-en.html
1- PhD Candidate of Linguistics, Allameh Tabataba’i University, Tehran, Iran
2- PhD in Psycholinguistics, Cluster of Languages of Emotion, Freie Universität Berlin, Berlin, Germany
2- PhD in Psycholinguistics, Cluster of Languages of Emotion, Freie Universität Berlin, Berlin, Germany
Abstract: (990 Views)
Introduction
Effective verbal communication is impossible without prosody, which transmits both linguistic and emotional information. This study aims to identify the brain regions involved in emotional prosody processing and uncover the mechanisms involved in understanding emotional prosody.
Methods
Using databases such as PubMed and Google Scholar, we searched for keywords such as emotional prosody, speech processing, event-related potentials (ERPs), and functional magnetic resonance imaging (fMRI) published between 2005 and 2021 on emotional prosody processing in healthy adult subjects’ brains. The search results for these keywords included more than 100 articles related to emotional prosody processing, an essential part of processing and understanding emotions. The subjects in all selected studies were healthy and right-handed regarding hearing, vision, and neuropsychology, and only studies using sentences as stimuli were included. The authors include only information regarding the present study questions in tables 1 and 2, and they avoid expressing technical details related to imaging and recording the brain signal, processing the images and signals, and avoiding issues related to statistical processing steps for the results of each study.
Results
Studying emotional prosody provided models for the stages of emotional tone processing. As one of these models that has been considered in subsequent research, the model of Schirmer and Kotz (11) was used to obtain more information about the brain areas involved in processing emotional speech sounds (2, 19). In the ear, brain stem, thalamus, and primary auditory cortex, speech sounds are processed and decoded in three stages. The first step is to extract phonological characteristics from prosodic cues in the right auditory cortex. In the second step, the right hemisphere’s posterior/posterior parts (STS) represent meaningful sequences of phonological elements. Evaluation and cognitive interpretation of expressed emotions constitute the third step, which produces simultaneous activity in two hemispheres of the frontal cortex (figures 1 and 2 show these steps).
Figure 1. Model for the processing of emotional prosody (11, p. 25)
Notably, according to the socio-cultural characteristics of their native language, the speakers use these phonological characteristics (such as fundamental frequency, intensity or loudness of the voice, speed of speech, and voice quality (breathy, whispering, sharp, growling, and the like) to produce and use emotional and attitudinal understanding (5, 24, 25).
Figure 2. Three successive steps for the processing of emotional prosody (20, p.261)
Conclusion
Processing speech sounds is confined to just one hemisphere of the brain. It is also crucial to understand that various factors, such as the difficulty level of the test, the quality of the stimuli, and the design and execution of the test, influence neural mechanisms. Given the functional complexity of linguistic prosody, analyzing the neural structure of emotional prosody seems more straightforward. Clinical and neurological research has shown that these two types of prosody process differently in the brain. Therefore, speech prosody should be examined according to its multi-level division and application rather than viewed as a general concept (5).
Given the intricate and multifaceted nature of language, it appears essential to analyze the emotional prosody of speech independently, owing to its complexity and diverse characteristics. Although it provides detailed instructions on how to create verbal-emotional communication, attention must also be paid to how syntax, meaning, and pragmatics influence the tone of speech (both emotional and linguistic).
For linguistic studies, particularly neurological studies, linguistic data are examined outside their natural context due to technical limitations. In order to obtain accurate information about how language is processed and understood, it is essential to carefully design the appropriate task and select and formulate the appropriate stimuli. Therefore, a standard form of emotional prosody in any language is necessary to research emotional prosody (8). As manipulated stimuli are usually used in these experiments, it is also necessary to identify the effective phonological characteristics (fundamental frequency, duration, intensity, and the like) associated with emotional, attitudinal, and motivational prosody in each language using appropriate perceptual and behavioral research.
In order to support researchers conducting proper research on Persian language processing, the steps involved in creating and maintaining the “Persian emotional speech database” will be explained. These databases can help researchers design and conduct neurological experiments based on ERP or fMRI techniques to investigate how the emotional prosody of Persian speech is processed and perceived in the brains of Persian speakers. The database was previously designed and produced at Freie Universität Berlin in collaboration with a research group (26). However, this information was only available in English. This study aimed to build an authentic database of emotional speech in Persian.
The database contains a set of 90 validated novel Persian sentences classified into five basic emotional categories (anger, disgust, fear, happiness, and sadness), as well as a neutral category. These sentences were validated in two experiments by a group of 1,126 native Persian speakers. The sentences were articulated by two native Persian speakers (one male, one female) in three conditions: 1) congruent (emotional lexical content articulated in a congruent emotional voice), 2) incongruent (neutral sentences articulated in an emotional voice), and 3) baseline (all emotional and neutral sentences articulated in neutral voice). The speech materials comprise about 470 sentences. The validity of the database was evaluated by a group of 34 native speakers in a perception test. Utterances recognized better than five times chance performance (71.4 %) were regarded as valid portrayals of the target emotions. Acoustic analysis of the valid emotional utterances revealed differences in pitch, intensity, and duration, attributes that may help listeners to correctly classify the intended emotion. The database is designed to be used as a reliable material source (for both text and speech) in future cross-cultural or cross-linguistic studies of emotional speech, and it is available for academic research purposes free of charge. This tool can benefit research in various fields, including neurology of language, psychology of language, clinical linguistics, speech therapy, and speech synthesis. To access the database, please contact the second author.
Ethical Considerations
Compliance with ethical guidelines
The present study has a review nature. Besides, there is no doubt that all the research introduced in this review had a valid code of ethics.
Authors’ contributions
First author: drafting the article, revising it, and being accountable for all aspects of the research. Second author: designing, constructing, and validating the Persian emotional speech database; writing the second part of the article; revising and correcting the entire article.
Funding
The German Research Foundation (DFG) awarded a scholarship to the second author for designing the Persian language emotional speech database.
Acknowledgments
Thanks to Shahla Raghibdoust for her assistance in writing this paper’s the first part (review).
Conflict of interest
The authors have no conflict of interest.
Effective verbal communication is impossible without prosody, which transmits both linguistic and emotional information. This study aims to identify the brain regions involved in emotional prosody processing and uncover the mechanisms involved in understanding emotional prosody.
Methods
Using databases such as PubMed and Google Scholar, we searched for keywords such as emotional prosody, speech processing, event-related potentials (ERPs), and functional magnetic resonance imaging (fMRI) published between 2005 and 2021 on emotional prosody processing in healthy adult subjects’ brains. The search results for these keywords included more than 100 articles related to emotional prosody processing, an essential part of processing and understanding emotions. The subjects in all selected studies were healthy and right-handed regarding hearing, vision, and neuropsychology, and only studies using sentences as stimuli were included. The authors include only information regarding the present study questions in tables 1 and 2, and they avoid expressing technical details related to imaging and recording the brain signal, processing the images and signals, and avoiding issues related to statistical processing steps for the results of each study.
Results
Studying emotional prosody provided models for the stages of emotional tone processing. As one of these models that has been considered in subsequent research, the model of Schirmer and Kotz (11) was used to obtain more information about the brain areas involved in processing emotional speech sounds (2, 19). In the ear, brain stem, thalamus, and primary auditory cortex, speech sounds are processed and decoded in three stages. The first step is to extract phonological characteristics from prosodic cues in the right auditory cortex. In the second step, the right hemisphere’s posterior/posterior parts (STS) represent meaningful sequences of phonological elements. Evaluation and cognitive interpretation of expressed emotions constitute the third step, which produces simultaneous activity in two hemispheres of the frontal cortex (figures 1 and 2 show these steps).
Figure 1. Model for the processing of emotional prosody (11, p. 25)
Notably, according to the socio-cultural characteristics of their native language, the speakers use these phonological characteristics (such as fundamental frequency, intensity or loudness of the voice, speed of speech, and voice quality (breathy, whispering, sharp, growling, and the like) to produce and use emotional and attitudinal understanding (5, 24, 25).
Figure 2. Three successive steps for the processing of emotional prosody (20, p.261)
Conclusion
Processing speech sounds is confined to just one hemisphere of the brain. It is also crucial to understand that various factors, such as the difficulty level of the test, the quality of the stimuli, and the design and execution of the test, influence neural mechanisms. Given the functional complexity of linguistic prosody, analyzing the neural structure of emotional prosody seems more straightforward. Clinical and neurological research has shown that these two types of prosody process differently in the brain. Therefore, speech prosody should be examined according to its multi-level division and application rather than viewed as a general concept (5).
Given the intricate and multifaceted nature of language, it appears essential to analyze the emotional prosody of speech independently, owing to its complexity and diverse characteristics. Although it provides detailed instructions on how to create verbal-emotional communication, attention must also be paid to how syntax, meaning, and pragmatics influence the tone of speech (both emotional and linguistic).
For linguistic studies, particularly neurological studies, linguistic data are examined outside their natural context due to technical limitations. In order to obtain accurate information about how language is processed and understood, it is essential to carefully design the appropriate task and select and formulate the appropriate stimuli. Therefore, a standard form of emotional prosody in any language is necessary to research emotional prosody (8). As manipulated stimuli are usually used in these experiments, it is also necessary to identify the effective phonological characteristics (fundamental frequency, duration, intensity, and the like) associated with emotional, attitudinal, and motivational prosody in each language using appropriate perceptual and behavioral research.
In order to support researchers conducting proper research on Persian language processing, the steps involved in creating and maintaining the “Persian emotional speech database” will be explained. These databases can help researchers design and conduct neurological experiments based on ERP or fMRI techniques to investigate how the emotional prosody of Persian speech is processed and perceived in the brains of Persian speakers. The database was previously designed and produced at Freie Universität Berlin in collaboration with a research group (26). However, this information was only available in English. This study aimed to build an authentic database of emotional speech in Persian.
The database contains a set of 90 validated novel Persian sentences classified into five basic emotional categories (anger, disgust, fear, happiness, and sadness), as well as a neutral category. These sentences were validated in two experiments by a group of 1,126 native Persian speakers. The sentences were articulated by two native Persian speakers (one male, one female) in three conditions: 1) congruent (emotional lexical content articulated in a congruent emotional voice), 2) incongruent (neutral sentences articulated in an emotional voice), and 3) baseline (all emotional and neutral sentences articulated in neutral voice). The speech materials comprise about 470 sentences. The validity of the database was evaluated by a group of 34 native speakers in a perception test. Utterances recognized better than five times chance performance (71.4 %) were regarded as valid portrayals of the target emotions. Acoustic analysis of the valid emotional utterances revealed differences in pitch, intensity, and duration, attributes that may help listeners to correctly classify the intended emotion. The database is designed to be used as a reliable material source (for both text and speech) in future cross-cultural or cross-linguistic studies of emotional speech, and it is available for academic research purposes free of charge. This tool can benefit research in various fields, including neurology of language, psychology of language, clinical linguistics, speech therapy, and speech synthesis. To access the database, please contact the second author.
Ethical Considerations
Compliance with ethical guidelines
The present study has a review nature. Besides, there is no doubt that all the research introduced in this review had a valid code of ethics.
Authors’ contributions
First author: drafting the article, revising it, and being accountable for all aspects of the research. Second author: designing, constructing, and validating the Persian emotional speech database; writing the second part of the article; revising and correcting the entire article.
Funding
The German Research Foundation (DFG) awarded a scholarship to the second author for designing the Persian language emotional speech database.
Acknowledgments
Thanks to Shahla Raghibdoust for her assistance in writing this paper’s the first part (review).
Conflict of interest
The authors have no conflict of interest.
Keywords: Emotional prosody, Event-related potential (ERP), Functional magnetic resonance imaging (fMRI), Persian emotional speech database (Persian ESD)
Received: 2023/11/9 | Accepted: 2024/06/6 | Published: 2024/08/28
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
