Research Article| Volume 37, ISSUE 3, P465.e27-465.e34, May 2023

Download started.


Patient-Reported Outcome Measures in Voice: An Updated Readability Analysis



      The purpose of this study was to investigate whether voice-related patient-reported outcome measures (PROMs) developed and validated since 2011 meet the recommendation by health literacy experts that such materials be written at a fifth-to-sixth grade reading level.


      A readability analysis of eight voice-related PROMs was conducted. Readability formulas utilized were the Coleman-Liau index, Flesch-Kincaid reading ease, FORCAST, simple measure of Gobbledygook index, and Gunning-Fog score.


      Three-fourths of the PROMs exceeded the recommended fifth- to sixth-grade reading level.


      Although awareness of health literacy has grown, voice-related PROMs continue to be developed without full consideration of their reading grade level. Researchers should consider revising or developing PROMs with consideration to reading grade level as well as other features to enhance readability.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Voice
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


      1. Center for American Progress. Improving literacy in the United States: Recommendations for increasing reading success. Accessed November 29, 2020.

        • Berkman ND
        • Sheridan SL
        • Donahue KE
        Low health literacy and health outcomes: an updated systematic review.
        Ann Intern Med. 2011; 155: 97-107
        • Smith SG
        • O'Conor R
        • Curtis LM
        • et al.
        Low health literacy predicts decline in physical function among older adults: findings from the LitCog cohort study.
        J Epidemiol Community Health. 2015; 69: 474-480
      2. Organization for Economic Cooperation and Development. OECD Skills Outlook 2013: First Results from the Survey of Adult Skills. Available at: Accessed October 29, 2020.

      3. Kutner, M, Greenberg, E, Baer, J. National Assessment of Adult Literacy (NAAL): A First Look at the Literacy of America's Adults in the 21st Century (Report No. NCES 2006-470). Available at: Accessed October 13, 2020.

      4. Institute of Medicine. Health Literacy: A Prescription to End Confusion. Available at: Accessed October 29, 2020.

      5. Ratzan, SC, Parker, RM. Health Literacy. Available at: Accessed October 13, 2020.

      6. Health Resources and Services Administration. Health Literacy. Available at:,to%20make%20appropriate%20health%20decisions. Accessed October 13, 2020.

        • Berkman ND
        • DeWalt DA
        • Pignone MP
        • et al.
        Literacy and health outcomes. Evidence Report/Technology Assessment No. 87 (Prepared by RTI International-University of North Carolina Evidence-based Practice Center under Contract No. 290-02-0016). AHRQ Publication No. 04-E007-2.
        Agency for Healthcare Research and Quality, Rockville, Maryland2004
        • Levy H.
        • Janke A.
        Health literacy and access to care.
        J Health Commun. 2016; 21: 43-50
        • Stewart MA.
        Effective physician-patient communication and health outcomes: a review.
        Canadian Med Assoc J. 1995; 152: 1423-1433
        • Zolnierek KB
        • Dimatteo MR.
        Physician communication and patient adherence to treatment: a meta-analysis.
        Med Care. 2009; 47: 826-834
        • DeWalt DA
        • Berkman ND
        • Sheridan S
        • et al.
        Literacy and health outcomes: a systematic review of the literature.
        J Gen Intern Med. 2004; 19: 1228-1239
        • Baker DW
        • Wolf MS
        • Feinglass J
        • et al.
        Health literacy and mortality among elderly persons.
        Arch Intern Med. 2007; 167: 1503-1509
        • Baker DW
        • Gazmararian JA
        • Williams MV
        • et al.
        Functional health literacy and the risk of hospital admission among Medicare managed care enrollees.
        J Gen Intern Med. 2002; 92: 1278-1283
        • Baker DW
        • Parker RM
        • Williams MV
        • et al.
        Health literacy and the risk of hospital admission.
        J Gen Intern Med. 1998; 13: 791-798
      7. U.S. Department of Health and Human Services. Health Communication. Available at: Accessed November 12, 2020.

        • Scientific Advisory Committee of the Medical Outcomes Trust (SACMOT)
        Assessing health status and quality-of-life instruments: attributes and review criteria.
        Qual Life Res. 2002; 11: 193-205
      8. DuBay, WH. The Principles of Readability. Available at: Accessed November 15, 2020.

        • Doak CC
        • Doak LG
        • Root JH
        Teaching Patients with Low Literacy Skills.
        1996 (Philadelphia, Pennsylvania)
      9. The Joint Commission. (2010). Advancing effective communication, cultural competence, and patient- and family-centered care: A roadmap for hospitals. Available at: Accessed November 15, 2020.

      10. Weiss, BD. Health Literacy: A Manual for Clinicians. Available at: Accessed November 11, 2020.

        • Weiss BD
        • Coyne CC
        Communicating with patients who cannot read.
        N Engl J Med. 1997; 337: 272-274
        • Azios JH
        • Bellon-Harn M
        • Dockens AL
        • et al.
        Quality and readability of English-language internet information for aphasia.
        Int J Speech-Lang Pathol. 2019; 21: 1-9
        • Dueppen AJ
        • Bellon-Harn ML
        • Radhakrishnan N.
        • et al.
        Quality and readability of English-language Internet information for voice disorders.
        J Voice. 2019; 33: 290-296
        • Gray SA
        • Zraick RI
        • Atcherson SR
        Readability of individuals with Disabilities Education Act Part B procedural safeguards: an update.
        Lang Speech Hear Serv Sch. 2019; 50: 373-384
        • Zraick RI
        • Atcherson SR
        Readability of patient-reported outcome questionnaires for use with persons with dysphonia.
        J Voice. 2012; 26: 635-641
        • Zraick RI
        • Atcherson SR
        • Brown AM
        Readability of patient-reported outcome questionnaires for use with persons who stutter.
        J Fluency. 2012; 37: 20-24
      11. ASHA. 2019 Member & Affiliate Profile. Available at: Accessed December 1, 2020.

      12. ASHA. About the American Speech-Language-Hearing Association (ASHA). Available at:Available at: Accessed December 1, 2020.

      13. National Quality Forum. Patient Reported Outcomes (PROs) in Performance Measurement. NQF: Patient-Reported Outcomes in Performance Measurement ( (Accessed 28 November 2020).

        • Atcherson SR
        • Richburg CM
        • Zraick RI
        • et al.
        Readability of questionnaires assessing listening difficulties associated with (central) auditory processing disorders.
        Lang Speech Hear Serv Sch. 2013; 44: 48-60
        • Douglas A
        • Kelly-Campbell RJ.
        Readability of patient-reported outcome measures in adult audiologic rehabilitation.
        Am J Audiol. 2018; 27: 173-247
        • Zraick RI
        • Atcherson SR
        • Ham BK
        Readability of patient-reported outcome measures for use with persons with swallowing disorders.
        Dysphagia. 2012; 27: 346-352
        • Coleman M
        • Liau TL.
        A computer readability formula designed for machine scoring.
        J Appl Psychol. 1975; 60: 283-284
        • Caylor JS
        • Sticht TG
        • Fox LC
        • et al.
        Methodologies for Determining Reading Requirements of Military Occupational Specialties. Tech. Report No. 73-5].
        1973 (Alexandria, VA)
        • Gunning R.
        The Technique of Clear Writing.
        1952 (New York)
        • McLaughlin G.
        SMOG grading: a new readability formula.
        J Read. 1969; 12: 639-646
        • Flesch R.
        A new readability yardstick.
        J Appl Psychol. 1948; 32: 221-233
      14. The U.S. Centers for Medicare & Medicaid Services. Toolkit for Making Written Material Clear and Effective. Available at: Accessed April 9, 2020.

        • Atcherson SR
        • Zraick RI
        • Brasseux RE
        Readability of patient-reported outcome questionnaires for use with persons with tinnitus.
        Ear Hear. 2011; 32: 671-673
        • Kahn A.
        • Pannbacker M.
        Readability of educational materials for clients with cleft lip/cleft palate and their families.
        Am J Speech Lang Pathol. 2000; 9: 3-9
        • Kelly-Campbell RJ
        • Zimmerman KR
        • Atcherson SR
        • et al.
        Readability of audiologic self-report assessment tools.
        J Acad Rehabil Audiol. 2012; 45: 63-73
        • Kramer JM
        • Schwartz A.
        Reducing barriers to patient-reported outcome measures for people with cognitive impairments.
        Arch Phys Med Rehabil. 2017; 98: 1705-1715
        • Paasche-Orlow MK
        • Wolf MS
        The causal pathways linking health literacy to health outcomes.
        Am J Health Behav. 2007; 31: S19-S26
        • Stormacq C
        • van den Broucke S
        • Wosinski J.
        Does health literacy mediate the relationship between socioeconomic status and health disparities? Integrative review.
        Health Promot Int. 2019; 34: e1-e17
        • Redish JC.
        Understanding the limitations of readability formulas.
        IEEE Trans Prof Commun. 1981; 24: 46-48
        • Zhou S
        • Jeong H
        • Green P.
        How consistent are the best-known readability equations in estimating the readability of design standards?.
        IEEE Trans Prof Commun. 2017; 60: 97-111
        • Friedman DB
        • Hoffman-Goetz L.
        A systematic review of readability and comprehension instruments used for print and web-based cancer information.
        Health Educ Behav. 2006; 33: 352-373
        • Hester EJ
        • Stevens-Ratchford R.
        Health literacy and the role of the speech-language pathologist.
        Am J Speech Lang Pathol. 2009; 18: 180-191
        • Davis TC
        • Wolf MS
        • Arnold CL
        • et al.
        Development and validation of the Rapid Estimate of Adolescent Literacy in Medicine (REALM-Teen): a tool to screen adolescents for below-grade reading in health care settings.
        Pediatr. 2006; 118: e1707-e1714
        • Parker RM
        • Baker DW
        • Williams MV
        • et al.
        The test of functional health literacy in adults.
        J Gen Intern Med. 1995; 10: 537-541
        • Morris N
        • MacLean CD
        • Chew LD
        • et al.
        The Single Item Literacy Screener: evaluation of a brief instrument to identify limited reading ability.
        BMC Fam Pract. 2006; 7: 21
        • Chew LD
        • Griffin JM
        • Partin MR
        • et al.
        Validation of screening questions for limited health literacy in a large VA outpatient population.
        J Gen Intern Med. 2008; 23: 561-566
        • Politi MC
        • Goodwin CM
        • Kaphingst KA
        • et al.
        How do subjective health literacy measures work in young adults? Specifying “Online” or “Paper-Based” forms impacts results.
        MDM Policy Pract. 2020; 5
        • Baker DW
        • Williams MV
        • Parker RM
        • et al.
        Development of a brief test to measure functional health literacy.
        Patient Educ Couns. 1999; 38: 33-42
        • Locklear T
        • Staman KL
        • Hudson KE
        Reaching Consensus on Patient-Centered Definitions: A Report from the Patient-Reported Outcomes PCORnet Task Force.
        NIH Collaboratory, 2015 (Available at:) (Accessed November 18, 2020)
      15. World Health Organization. International Classification of Functioning, Disability, and Health (ICF). Available at: Accessed December 6, 2020.

        • Rothrock N
        • Kaiser K
        • Cella D.
        Developing a valid patient-reported outcome measure.
        Clin Pharmacol Ther. 2011; 90: 737-742
        • Rademakers J
        • Heijmans M.
        Beyond reading and understanding: health literacy as the capacity to act.
        Int J Environ Res Pub Health. 2018; 15: 1676
      16. Pearson. Universal Design for Computer-Based Testing (UD-CBT) Guidelines. Available at: Accessed December 15, 2020.

      17. Beddow, PA, Ellion SN, Kettler, RJ. Test Accessibility: Item Reviews and Lessons Learned from Four State Assessments. Available at: Accessed December 15, 2020.

        • Abedi J
        • Leon S
        • Kao J
        • et al.
        Accessible Reading Assessments for Students with Disabilities: The Role of Cognitive, Grammatical, Lexical and Textual/Visual Features.
        National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Los Angeles, CA2011 (CRESST Report 785)
        • Kettler RJ.
        Testing accommodations: theory and research to inform practice.
        Int J Disabil Dev Educ. 2012; 59: 53-66
        • Nind M
        • Seale J.
        Concepts of access for people with learning difficulties: towards a shared understanding.
        Disabil Soc. 2009; 24: 273-287
        • Hartley SL
        • MacLean Jr, WE
        A review of the reliability and validity of Likert-type scales for people with intellectual disability.
        J Intellect Disabil Res. 2006; 50: 813-827
        • Boone DR
        • McFarlane SC
        • von Berg S
        • et al.
        The Voice and Voice Therapy.
        Pearson Education, 2020 (10th ed.)
        • National Research Council
        Division of Behavioral and Social Sciences and Education. Commission on Behavioral and Social Sciences and Education. Committee on National Statistics. Cognitive Aspects of Survey Methodology: Building a bridge between disciplines.
        National Academies Press, 1984
        • Fujiura GT
        • Expert RRTC
        Panel on Health Measurement. Self-reported health of people with intellectual disability.
        Intellect Dev Disabil. 2012; 50: 352-369
        • Kocher R
        • Emanuel EJ
        • DeParle NAM
        The Affordable Care Act and the future of clinical medicine: the opportunities and challenges.
        Ann Intern Med. 2010; 153: 536-539
        • Etter NM
        • Hapner ER
        • Barkmeier-Kraemer JM
        • et al.
        Aging Voice Index (AVI): reliability and validity of a voice quality of life scale for older adults.
        J Voice. 2019; 33 (807.e8)
        • Ghirardi ACAM
        • Ferreira LP
        • Giannini SPP
        • et al.
        Screening Index for Voice Disorder (SIVD): development and validation.
        J Voice. 2013; 27: 195-200
        • Naunheim MR
        • Dai JB
        • Rubinstein BJ
        • et al.
        A visual analog scale for patient-reported voice outcomes: the VAS voice.
        Laryng Investig Otolaryngol. 2019; 5
        • Epstein R
        • Hirani SP
        • Stygall J
        • et al.
        How do individuals cope with voice disorders? Introducing the voice disability coping questionnaire.
        J Voice. 2009; 23: 209-217
        • Paolillo NP
        • Pantaleo G.
        Development and validation of the Voice Fatigue Handicap Questionnaire (VFHQ): clinical, psychometric, and psychosocial facets.
        J Voice. 2015; 25: 91-100
        • Nanjundeswaran C
        • Jacobson BH
        • Gartner-Schmidt J
        • et al.
        Vocal Fatigue Index (VFI): development and validation.
        J Voice. 2015; 29: 433-440
        • Lehto L
        • Rantala L
        • Vilkman E
        • et al.
        Experiences of a short vocal training course for call-center customer service advisors.
        Fol Phoniatr Logop. 2003; 55: 163-176
        • Isetti D
        • Meyer T.
        Workplace productivity and voice disorders: a cognitive interviewing study on presenteeism in individuals with spasmodic dysphonia.
        J Voice. 2014; 28: 700-710