Literature Reviews

Reviewed research papers, articles, and publications relevant to digital accessibility.

• WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics

  Jason Wu, Siyan Wang, Siman Shen, Yi-Hao Peng, Jeffrey Nichols, Jeffrey P. Bigham (2023) — Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

  This paper introduces WebUI, a large-scale dataset of approximately 400,000 web pages automatically crawled and paired with visual, semantic, and stylistic metadata extracted from the browser engine. The dataset addresses a critical bottleneck in UI …

  • machine learning
  • computer vision
  • UI modeling
  • web semantics
  • transfer learning
  • datasets
  • mobile accessibility
  • accessibility metadata

• Never-ending Learning of User Interfaces

  Jason Wu, Rebecca Krosnick, Eldon Schoop, Amanda Swearngin, Jeffrey P. Bigham, Jeffrey Nichols (2023) — Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST)

  This paper introduces the Never-ending UI Learner, an automated system that continuously crawls real mobile applications to learn semantic properties of user interfaces. The system addresses a fundamental limitation of current approaches to UI unders…

  • machine learning
  • mobile accessibility
  • UI semantics
  • automated crawling
  • tappability
  • screen readers
  • computer vision

• NoTeeline: Supporting Real-Time, Personalized Notetaking with LLM-Enhanced Micronotes

  Faria Huq, Abdus Samee, David Chuan-En Lin, Alice Xiaodi Tang, Jeffrey P. Bigham (2025) — Proceedings of the 30th International Conference on Intelligent User Interfaces (IUI '25)

  This paper introduces NoTeeline, an interactive notetaking tool that uses LLMs to expand user-written "micronotes" — brief shorthand jottings like "plastic pol. ->" or "RNNs are unrolled l to r or opp" — into full-fledged notes that maintain the user…

  • large language models
  • writing assistance
  • personalization
  • notetaking
  • cognitive load
  • education
  • video accessibility
  • user agency

• Technical Perspective: Computation Where the (Inter)Action Is

  Jeffrey P. Bigham (2022) — Communications of the ACM

  This one-page technical perspective accompanies the SoundWatch paper in Communications of the ACM. Bigham uses SoundWatch — a smartwatch prototype that detects audio events and displays descriptions for deaf and hard-of-hearing people — as a lens to …

  • wearable technology
  • deaf and hard of hearing
  • sound recognition
  • smartwatch
  • assistive technology
  • machine learning
  • edge computing
  • accessibility

• Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors

  Michelle S. Lam, Fred Hohman, Dominik Moritz, Jeffrey P. Bigham, Kenneth Holstein, Mary Beth Kery (2025) — Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST '25)

  This paper introduces "policy maps," an approach to AI policy design for large language models inspired by physical mapmaking. The core insight is that comprehensive policy coverage over an unbounded space of LLM inputs and outputs is impossible — ju…

  • AI safety
  • AI policy
  • large language models
  • AI ethics
  • model evaluation
  • content moderation
  • data visualization
  • human-AI interaction
  • AI governance

• UIClip: A Data-driven Model for Assessing User Interface Design

  Jason Wu, Yi-Hao Peng, Xin Yue Amanda Li, Amanda Swearngin, Jeffrey P. Bigham, Jeffrey Nichols (2024) — Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST '24)

  This paper introduces UIClip, a computational model that automatically assesses UI design quality and visual relevance from a screenshot and natural language description. Built on OpenAI's CLIP B/32 architecture (151 million parameters), UIClip is fi…

  • UI design
  • machine learning
  • design quality assessment
  • computer vision
  • CLIP
  • visual design
  • design tools
  • user interface
  • automated evaluation

• CodeA11y: Making AI Coding Assistants Useful for Accessible Web Development

  Peya Mowar, Yi-Hao Peng, Jason Wu, Aaron Steinfeld, Jeffrey P. Bigham (2025) — Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25)

  This paper addresses a persistent problem: despite decades of accessibility standards and tools, ~96% of web pages contain accessibility violations. The authors argue that AI coding assistants like GitHub Copilot represent an untapped opportunity bec…

  • web accessibility
  • AI coding assistants
  • developer tools
  • WCAG
  • automated testing
  • GitHub Copilot
  • large language models
  • accessibility education
  • software development

• Exploring the Role of Social Support When Integrating Generative AI in Small Business Workflows

  Quentin Romero Lauro, Jeffrey P. Bigham, Yasmine Kotturi (2024) — Companion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing (CSCW Companion '24)

  This paper investigates how small business owners from resource-constrained communities leverage their offline social networks to integrate generative AI technologies into their business workflows. The researchers interviewed 11 participants — seven …

  • generative AI
  • entrepreneurship
  • social support
  • technology adoption
  • digital divide
  • digital literacy
  • lean economies
  • human-computer interaction

• StepWrite: Adaptive Planning for Speech-Driven Text Generation

  Hamza El Alaoui, Atieh Taheri, Yi-Hao Peng, Jeffrey P. Bigham (2025) — Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST '25)

  This paper introduces StepWrite, an LLM-powered voice-based writing system that enables structured, hands-free and eyes-free composition of longer-form texts. While speech-to-text tools handle short dictation well, composing structured emails or deta…

  • voice interface
  • speech-to-text
  • hands-free interaction
  • eyes-free interaction
  • large language models
  • text composition
  • cognitive load
  • adaptive planning
  • writing assistance
  • motor impairment
  • multitasking

• Morae: Proactively Pausing UI Agents for User Choices

  Yi-Hao Peng, Dingzeyu Li, Jeffrey P. Bigham, Amy Pavel (2025) — Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST '25)

  This paper introduces Morae, a UI agent that proactively pauses during automated task execution to involve blind and low-vision (BLV) users in critical decisions, rather than completing tasks end-to-end without user input. The work is motivated by a …

  • UI agents
  • blind and low vision
  • large language models
  • human-agent interaction
  • user agency
  • mixed-initiative interaction
  • web accessibility
  • screen readers
  • artificial intelligence