Literature Reviews

Reviewed research papers, articles, and publications relevant to digital accessibility.

• Remotely Co-Designing Features for Communication Applications using Automatic Captioning with Deaf and Hearing Pairs

  Matthew Seita, Sooyeon Lee, Sarah Andrew, Kristen Shinohara, Matt Huenerfauth (2022) — Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22)

  This CHI 2022 paper addresses two intertwined problems. First, methodologically, how can co-design research involving both Deaf/Hard-of-Hearing (DHH) and hearing participants be conducted remotely during and beyond COVID-19, when in-person sessions a…

  • automatic speech recognition
  • deaf and hard of hearing
  • participatory design
  • co-design
  • videoconferencing
  • captioning
  • captions
  • mixed-ability
  • accessibility research
  • remote research

• Deaf Individuals' Views on Speaking Behaviors of Hearing Peers when Using an Automatic Captioning App

  Matthew Seita, Matt Huenerfauth (2020) — Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA '20)

  This CHI 2020 Late-Breaking Work paper investigates what behaviors hearing speakers should ideally exhibit when holding in-person conversations with Deaf or deaf people using an Automatic Speech Recognition (ASR) captioning app on a mobile device. Th…

  • automatic speech recognition
  • deaf and hard of hearing
  • captioning
  • captions
  • speaking behavior
  • mobile accessibility
  • communication
  • user research
  • accessibility research

• Caption Royale: Exploring the Design Space of Affective Captions from the Perspective of Deaf and Hard-of-Hearing Individuals

  Caluã de Lacerda Pataca, Saad Hassan, Nathan Tinker, Roshan Lalintha Peiris, Matt Huenerfauth (2024) — CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

  This CHI 2024 paper from Rochester Institute of Technology and Tulane tackles a concrete design question: if we want captions to convey a speaker's emotion — not just their words — which typographic modulations should we use? Prior work had establish…

  • captioning
  • affective captions
  • expressive captions
  • typography
  • valence
  • arousal
  • deaf and hard of hearing
  • DHH
  • circumplex model
  • variable fonts
  • best-worst scaling
  • accessibility design

• Fuzzy Feelings: Arousal's Interpretive Noise and the Case for Acoustic-Based Haptics

  Caluã de Lacerda Pataca, Stephanie Patterson, Roshan L Peiris, Matt Huenerfauth (2026) — CHI '26: Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems

  This CHI 2026 paper from a team at Rochester Institute of Technology and Birmingham City University tackles a persistent gap in captioning: traditional captions carry words but strip the emotional tone, rhythm, and vocal affect that sighted hearing v…

  • captioning
  • expressive captions
  • haptic feedback
  • vibrotactile
  • affective computing
  • speech emotion recognition
  • deaf and hard of hearing
  • DHH
  • arousal
  • valence
  • multimodal accessibility
  • prosody

• Multiple View Perspectives: Improving Inclusiveness and Video Compression in Mainstream Classroom Recordings

  Raja S. Kushalnagar, Anna C. Cavender, Jehan-François Pâris (2010) — Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2010)

  This paper presents Multiple View Perspectives (MVP), a system that captures and presents multiple focused video views of a classroom for deaf and hard of hearing (DHH) students. DHH students in mainstream classrooms face a fundamental visual attenti…

  • deaf and hard of hearing
  • lecture capture
  • video accessibility
  • classroom accessibility
  • sign language interpreting
  • captioning
  • visual attention
  • video compression
  • inclusive education

• Crowd Caption Correction (CCC)

  Rebecca Perkins Harrington, Gregg C. Vanderheiden (2013) — Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '13)

  This short paper presents Crowd Caption Correction (CCC), a feature that allows meeting participants or authorized third parties to correct errors in real-time captions during telecollaboration sessions. Captions are critical for deaf and hard of hea…

  • captioning
  • deaf and hard of hearing
  • crowdsourcing
  • telecollaboration
  • real-time captioning
  • universal design
  • open source
  • transcription
  • videoconferencing

• Real-Time Captioning by Non-Experts with Legion Scribe

  Walter S. Lasecki, Christopher D. Miller, Raja Kushalnagar, Jeffrey P. Bigham (2013) — Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2013)

  This short paper introduces Legion Scribe (Scribe), a system that enables 3-5 non-expert typists to collectively caption speech in real time, achieving accuracy approaching that of a professional stenographer at 20-30% of the cost. The system address…

  • real-time captioning
  • deaf and hard of hearing
  • crowdsourcing
  • human computation
  • assistive technology
  • captioning

• Scribe: Deep Integration of Human and Machine Intelligence to Caption Speech in Real Time

  Walter S. Lasecki, Christopher D. Miller, Iftekhar Naim, Raja Kushalnagar, Adam Sadilek, Daniel Gildea, Jeffrey P. Bigham (2017) — Communications of the ACM

  Scribe is a system that provides on-demand, real-time captioning of live speech for deaf and hard of hearing (DHH) people by combining groups of non-expert human captionists with machine intelligence. The system addresses a critical accessibility gap…

  • real-time captioning
  • deaf and hard of hearing
  • crowdsourcing
  • human computation
  • speech recognition
  • assistive technology
  • captioning

• Leveraging Complementary Contributions of Different Workers for Efficient Crowdsourcing of Video Captions

  Yun Huang, Yifeng Huang, Na Xue, Jeffrey P. Bigham (2017) — CHI Conference on Human Factors in Computing Systems

  This paper presents BandCaption, a crowdsourcing system that combines automatic speech recognition (ASR) with input from diverse crowd workers to efficiently correct video captions. The key insight is that different groups of people — hearing-impaire…

  • captioning
  • crowdsourcing
  • video accessibility
  • speech recognition
  • deaf and hard of hearing
  • second language learners
  • automatic speech recognition

• Enhancing Caption Accessibility through Simultaneous Multimodal Information: Visual-Tactile Captions

  Raja S. Kushalnagar, Gary W. Behm, Joseph S. Stanislow, Vasu Gupta (2014) — Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS)

  This paper addresses a fundamental limitation of captions (subtitles) for deaf and hard of hearing (DHH) viewers: captions force viewers to split attention between reading text at the bottom of the screen and watching the visual action, inevitably ca…

  • captioning
  • deaf and hard of hearing
  • haptic feedback
  • multimodal interaction
  • non-speech information
  • sensory substitution
  • multimedia accessibility