Research Article|Articles in Press

Comparative Analysis of Two Methods of Perceptual Voice Assessment



      The primary aim was to compare two methods for perceptual evaluation of voice – paired comparison (PC) and visual analog scale (VAS) ratings. Secondary aims were to assess the correspondence between two dimensions of voice– overall severity of voice quality and resonant voice, and to investigate the influence of rater experience on perceptual rating scores and rating confidence scores.

      Study Design

      Experimental design.


      Voice samples from six children (pre and post therapy) were rated by 15 Speech-Language Pathologists specialized in voice. Raters completed four tasks corresponding to the two rating methods and voice qualities: PC-severity, PC-resonance, VAS-severity, and VAS-resonance. For PC tasks, raters chose the better of two voice samples (better voice quality or better resonance, depending on the task) and indicated the degree of confidence in each choice. Rating and confidence score were combined to produce a number on a 1-10 scale (PC-confidence adjusted). VAS ratings involved rating voices on a scale for degree of severity and resonance, respectively.


      PC-confidence adjusted and VAS ratings were moderately correlated for overall severity and also vocal resonance. VAS ratings were normally distributed and had greater rater consistency than PC-confidence adjusted ratings. VAS scores reliably predicted binary PC choices (choice of voice sample only). Overall severity and vocal resonance were weakly correlated and rater experience was not linearly related to rating scores or confidence.


      Results suggest that the VAS rating method holds advantages over PC, including normally distributed ratings, superior consistency of ratings, and the ability to provide more finely grained detail regarding the auditory perception of voice. Overall severity and vocal resonance were not redundant in the current data set, suggesting that resonant voice and overall severity are not isomorphic. Finally, the number of years of clinical experience was not linearly related to perceptual ratings or rating confidence.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Voice
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Watts CR
        • Awan SN.
        Laryngeal Function and Voice Disorders: Basic Science to Clinical Practice.
        Thieme Publishers, New York, NY2019
        • Kreiman J
        • Gerratt BR.
        Defining and studying voice across disciplinary boundaries.
        in: Eidsheim N Meizel K The Oxford Handbook of Voice Studies. Oxford Handbooks, New York, NY2019: 493-514
        • Kreiman J
        • Sidtis D.
        Foundations of Voice Studies: An Interdisciplinary Approach to Voice Production and Perception.
        John Wiley & Sons, Chichester, UK2011
        • Kreiman J
        • Gerratt BR
        • Kempster GB
        • et al.
        Perceptual evaluation of voice quality: review, tutorial, and a framework for future research.
        J Speech Hear Res. 1993; 36: 21-40
        • Fujiki RB
        • Thibeault SL.
        The relationship between auditory-perceptual rating scales and objective voice measures in children with voice disorders.
        Am J Speech Lang Pathol. 2021; 30: 228-238
        • Kelchner LN
        • Brehm SB
        • Weinrich B
        • et al.
        Perceptual evaluation of severe pediatric voice disorders: rater reliability using the consensus auditory perceptual evaluation of voice.
        J Voice. 2010; 24: 441-449
        • Boseley ME
        • Hartnick CJ.
        Development of the human true vocal fold: depth of cell layers and quantifying cell types within the lamina propria.
        Ann Otol Rhinol Laryngol. 2006; 115: 784-788
      1. Döllinger M, Dubrovskiy D, Patel R. Spatiotemporal analysis of vocal fold vibrations between children and adults. the Laryngoscope. 2012;122:2511-2518.

        • Eckel HE
        • Sprinzl GM
        • Koebke J
        • et al.
        Morphology of the human larynx during the first five years of life studied on whole organ serial sections.
        Ann Otol Rhinol Laryngol. 1999; 108: 232-238
        • Kahane JC.
        A morphological study of the human prepubertal and pubertal larynx.
        Am J Anat. 1978; 151: 11-19
        • Kerschner JE
        • Merati AL.
        Science of voice production from infancy through adolescence.
        in: Hartnick CJ Boseley ME Pediatric Voice Disorders. Plural Publishing, San Diego, CA2008: 23-29
        • Litman RS
        • Weissend EE
        • Shibata D
        • et al.
        Developmental changes of laryngeal dimensions in unparalyzed.
        Sedated Child: Anesthesiol. 2003; 98: 41-45
        • Patel RR.
        Vibratory onset and offset times in children: A laryngeal imaging study.
        Int J Pediatr Otorhinolaryngol. 2016; 87: 11-17
        • Patel RR
        • Dubrovskiy D
        • Dollinger M.
        Measurement of glottal cycle characteristics between children and adults: physiological variations.
        J Voice Off J Voice Found. 2014; 28: 476-486
        • Patel RR
        • Donohue KD
        • Unnikrishnan H
        • et al.
        Kinematic measurements of the vocal-fold displacement waveform in typical children and adult populations: quantification of high-speed endoscopic videos.
        J Speech Lang Hear Res. 2015; 58: 227-240
        • Soltani M
        • Ashayeri H
        • Modarresi Y
        • et al.
        Fundamental frequency changes of persian speakers across the life span.
        J Voice. 2014; 28: 274-281
        • Spazzapan EA
        • Marino VC de C
        • Cardoso VM
        • et al.
        Acoustic characteristics of voice in different cycles of life: an integrative literature review.
        Rev CEFAC. 2019; 21: e15018
        • Spazzapan EA
        • Fabron EMG
        • Berti LC
        • Chagas EFB
        • Marino VC de C.
        Acoustic characteristics of the voice for Brazilian Portuguese speakers across the life span.
        J Voice. 2020; (Published online October)S0892199720303635
        • Stathopoulos ET
        • Huber JE
        • Sussman JE.
        Changes in acoustic characteristics of the voice across the life span: measures from individuals 4–93 years of age.
        J Speech Lang Hear Res. 2011; 54: 1011-1021
        • Law T
        • Kim JH
        • Lee KY
        • et al.
        Comparison of rater's reliability on perceptual evaluation of different types of voice sample.
        J Voice. 2012; 26 (666.e13-666.e21)
        • Zraick RI
        • Wendel K
        • Smith-Olinde L
        The effect of speaking task on perceptual judgment of the severity of dysphonic voice.
        J Voice. 2005; 19: 574-581
        • Kreiman J
        • Gerratt BR
        • Precoda K
        • et al.
        Individual differences in voice quality perception.
        J Speech Lang Hear Res. 1992; 35: 512-520
        • Kreiman J
        • Gerratt BR.
        Validity of rating scale measures of voice quality.
        J Acoust Soc Am. 1998; 104: 1598-1608
        • Martins PC
        • Couto TE
        • Gama ACC
        Auditory-perceptual evaluation of the degree of vocal deviation: correlation between the visual analogue scale and numerical scale.
        CoDAS. 2015; 27: 279-284
        • Webb AL
        • Carding PN
        • Deary IJ
        • et al.
        The reliability of three perceptual evaluation scales for dysphonia.
        Eur Arch Otorhinolaryngol. 2004; 261: 429-434
        • Wuyts FL
        • De Bodt MS
        • Van de Heyning PH
        Is the reliability of a visual analog scale higher than an ordinal scale? An experiment with the GRBAS scale for the perceptual evaluation of dysphonia.
        J Voice. 1999; 13: 508-517
        • Yamasaki R
        • Madazio G
        • Leão SHS
        • et al.
        Auditory-perceptual evaluation of normal and dysphonic voices using the voice deviation scale.
        J Voice. 2017; 31: 67-71
        • Kreiman J
        • Gerratt BR
        • Berke GS.
        The multidimensional nature of pathologic vocal quality.
        J Acoust Soc Am. 1994; 96: 1291-1302
        • Kreiman J
        • Gerratt BR.
        The perceptual structure of pathologic voice quality.
        J Acoust Soc Am. 1996; 100: 1787-1795
        • Yiu EM-L
        • Chan KMK
        Mok RS -M. Reliability and confidence in using a paired comparison paradigm in perceptual voice quality evaluation.
        Clin Linguist Phon. 2007; 21: 129-145
        • Gerratt BR
        • Kreiman J
        • Antonanzas-Barroso N
        • et al.
        Comparing internal and external standards in voice quality judgments.
        J Speech Lang Hear Res. 1993; 36: 14-20
        • Gerratt BR
        • Kreiman J.
        Measuring vocal quality with speech synthesis.
        J Acoust Soc Am. 2001; 110: 7
        • Rabinov CR
        • Kreiman J
        • Gerratt BR
        • et al.
        Comparing reliability of perceptual ratings of roughness and acoustic measures of jitter.
        J Speech Lang Hear Res. 1995; 38: 26-32
        • Awan SN
        • Roy N
        • Jetté ME
        • et al.
        Quantifying dysphonia severity using a spectral/cepstral-based acoustic index: Comparisons with auditory-perceptual judgements from the CAPE-V.
        Clin Linguist Phon. 2010; 24: 742-758
        • Awan SN
        • Lawson LL.
        The effect of anchor modality on the reliability of vocal severity ratings.
        J Voice. 2009; 23: 341-352
        • Gaskill CS
        • Awan JA
        • Watts CR
        • et al.
        Acoustic and perceptual classification of within-sample normal, intermittently dysphonic, and consistently dysphonic voice types.
        J Voice. 2017; 31: 218-228
        • Kreiman J
        • Gerratt BR.
        Sources of listener disagreement in voice quality assessment.
        J Acoust Soc Am. 2000; 108: 1867-1876
        • Ghio A
        • Revis J
        • Smithson-Barrière D
        • et al.
        Reliability and correlations between overall severity, roughness and breathiness in the perception of dysphonic voices: investigating cognitive aspects.
        J Voice. 2021; (Published online August)S0892199721002460
        • Walden PR
        • Rau S.
        Individual voice dimensions’ prediction of overall dysphonia severity on two auditory-perceptual scales.
        J Speech Lang Hear Res. 2022; 65: 2759-2777
        • Berry DA
        • Verdolini K
        • Montequin DW
        • et al.
        A quantitative output-cost ratio in voice production.
        J Speech Lang Hear Res. 2001; 44: 29-37
        • Peterson KL
        • Verdolini-Marston K
        • Barkmeier JM
        • et al.
        Comparison of aerodynamic and electroglottographic parameters in evaluating clinically relevant voicing patterns.
        Ann Otol Rhinol Laryngol. 1994; 103: 335-346
        • Verdolini-Marston K
        • Katherine Burke M
        • Lessac A
        • et al.
        Preliminary study of two methods of treatment for laryngeal nodules.
        J Voice. 1995; 9: 74-85
        • De Bodt MS
        • Wuyts FL
        • Van de Heyning PH
        • et al.
        Test-retest study of the GRBAS scale: Influence of experience and professional background on perceptual rating of voice quality.
        J Voice. 1997; 11: 74-80
        • Eadie TL
        • Nicolici C
        • Baylor C
        • et al.
        Effect of experience on judgments of adductor spasmodic dysphonia.
        Ann Otol Rhinol Laryngol. 2007; 116: 695-701
        • Eadie TL
        • Kapsner M
        • Rosenzweig J
        • et al.
        The role of experience on judgments of dysphonia.
        J Voice. 2010; 24: 564-573
        • Damrose JF
        • Goldman SN
        • Groessl EJ
        • et al.
        The impact of long-term botulinum toxin injections on symptom severity in patients with spasmodic dysphonia.
        J Voice. 2004; 18: 415-422
        • Kreiman J
        • Gerratt BR
        • Precoda K.
        Listener experience and perception of voice quality.
        J Speech Hear Res. 1990; 33: 103-115
        • Eadie TL
        • Kapsner-Smith M.
        The effect of listener experience and anchors on judgments of dysphonia.
        J Speech Lang Hear Res. 2011; 54: 430-447
        • Verdolini Abbott K
        • Hersan R
        • Hammer D
        • et al.
        Adventures in Voice: A Whole New Way of Doing Things for Kids Clinician Manual.
        3rd ed. Visions in Voice, Lawrence, KS2015
        • Feinstein H
        • Daşdöğen Ü
        • Libertus ME
        • et al.
        Cognitive mechanisms in pediatric voice therapy – an initial examination.
        J Voice. 2021; (Published online)S0892199721003246
        • Kempster GB
        • Gerratt BR
        • Verdolini Abbott K
        • et al.
        Consensus auditory-perceptual evaluation of voice: development of a standardized clinical protocol.
        Am J Speech Lang Pathol. 2009; 18: 124-132
        • Harris PA
        • Taylor R
        • Minor BL
        • et al.
        The REDCap consortium: building an international community of software platform partners.
        J Biomed Inform. 2019; 95103208
        • Schober P
        • Boer C
        • Schwarte LA.
        Correlation coefficients: appropriate use and interpretation.
        Anesth Analg. 2018; 126: 1763-1768
        • Koo TK
        • Li MY.
        A guideline of selecting and reporting intraclass correlation coefficients for reliability research.
        J Chiropr Med. 2016; 15: 155-163
        • Hosmer Jr., DW
        • Lemeshow S.
        Applied Logistic Regression.
        Wiley-Interscience, Chichester, UK2001
        • Verdolini Abbott K.
        Some guiding principles in emerging models of voice therapy for children.
        Semin Speech Lang. 2013; 34: 080-093
        • McCullough G
        • Pickett H
        • Balou S
        • et al.
        Treatment of laryngeal hyperfunction with flow phonation: a pilot study.
        J Laryngol Voice. 2012; 2: 64
        • Watts CR
        • Hamilton A
        • Toles L
        • et al.
        A randomized controlled trial of stretch-and-flow voice therapy for muscle tension dysphonia.
        The Laryngoscope. 2015; 125: 1420-1425
        • Watts CR
        • Hamilton A
        • Toles L
        • et al.
        Intervention outcomes of two treatments for muscle tension dysphonia: a randomized controlled trial.
        J Speech Lang Hear Res. 2019; 62: 272-282
        • American Speech-Language-Hearing Association
        2017 SLP Health Care Survey: Caseload Characteristics.
        2017 (Accessed June 30, 2020)
        • American Speech-Language-Hearing Association
        2018 Schools Survey: SLP Caseload and Workload Characteristics.
        2018 (Accessed June 30, 2020)
        • American Speech-Language-Hearing Association
        2019 SLP Health Care Survey: Caseload Characteristics.
        2019 (Accessed June 30, 2020)
        • American Speech-Language-Hearing Association
        2020 Schools Survey: SLP Caseload and Workload Characteristics.
        2020 (Accessed August 24, 2022)
        • American Speech-Language-Hearing Association
        2022 Schools Survey: SLP Caseload and Workload Characteristics.
        2022 (Accessed August 24, 2022)
        • Chan KMK
        • Yiu EML.
        A comparison of two perceptual voice evaluation training programs for naive listeners.
        J Voice. 2006; 20: 229-241
        • Eddins DA
        • Anand S
        • Lang A
        • et al.
        Developing clinically relevant scales of breathy and rough voice quality.
        J Voice. 2021; 35 (663.e9-663.e16)