Literature Reviews

Reviewed research papers, articles, and publications relevant to digital accessibility.

• SAMBA: a semi-automatic method for measuring barriers of accessibility

  Giorgio Brajnik, Raffaella Lomuscio (2007) — Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '07)

  Brajnik and Lomuscio argue that web accessibility cannot be managed without being measured, yet existing accessibility metrics are underdeveloped and almost universally tied to conformance with WCAG checkpoints rather than to the real-world barriers …

  • web accessibility
  • accessibility metrics
  • accessibility evaluation
  • accessibility testing
  • conformance testing
  • barrier walkthrough
  • expert review
  • automated testing
  • quality assurance

• Effects of sampling methods on web accessibility evaluations

  Giorgio Brajnik, Andrea Mulas, Claudia Pitton (2007) — Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '07)

  Brajnik, Mulas, and Pitton's Assets '07 paper is a large, careful empirical investigation of the sampling step in web-accessibility evaluation — the step at which an evaluator picks which pages of a site to actually test. For any site above a trivial…

  • web accessibility
  • accessibility evaluation
  • sampling methods
  • accessibility metrics
  • conformance testing
  • automated testing
  • WCAG
  • quality assurance
  • research methodology
  • accessibility audit

• A comparative test of web accessibility evaluation methods

  Giorgio Brajnik (2008) — Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '08)

  This paper presents a controlled laboratory experiment comparing two web accessibility evaluation methods: conformance review (CR) and barrier walkthrough (BW). Conformance review is the traditional approach where evaluators check whether a page sati…

  • accessibility evaluation
  • conformance testing
  • barrier walkthrough
  • web accessibility
  • WCAG
  • evaluation methods
  • accessibility testing
  • quality assessment

• Validity and Reliability of Web Accessibility Guidelines

  Giorgio Brajnik (2009) — Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '09)

  This paper presents an experiment measuring the validity and reliability of WCAG 1.0 and WCAG 2.0 checkpoints when applied by human evaluators. The study recruited 35 young web developers with some accessibility knowledge from a university course and…

  • accessibility evaluation
  • WCAG
  • conformance testing
  • inter-rater reliability
  • web accessibility
  • guidelines

• Is Accessibility Conformance an Elusive Property? A Study of Validity and Reliability of WCAG 2.0

  Giorgio Brajnik, Yeliz Yesilada, Simon Harper (2012) — ACM Transactions on Accessible Computing

  This landmark study investigates whether WCAG 2.0's definition of "Reliably Human Testable"—that at least 80% of knowledgeable evaluators would agree on an audit conclusion—is actually achievable in practice. The researchers recruited 25 experienced …

  • WCAG
  • conformance testing
  • accessibility evaluation
  • evaluator effect
  • reliability
  • validity
  • expert review

• Guidelines, Icons and Marketable Skills: An Accessibility Evaluation of 100 Web Development Company Homepages

  Teresa D. Gilbertson, Colin H. C. Machin (2012) — Proceedings of the International Cross-Disciplinary Conference on Web Accessibility (W4A)

  This study examined whether web development companies that market accessibility as a skill actually practice what they preach, by evaluating the homepages of 100 UK web development companies selected from Google searches across four geographic region…

  • web accessibility
  • WCAG compliance
  • developer awareness
  • conformance testing
  • accessibility evaluation
  • web development
  • accessibility policy

• Certification or Conformance: Making a Successful Commitment to WCAG 2.0

  Suzette Keith, Nikolaos Floratos, Gill Whitney (2012) — Proceedings of the International Cross-Disciplinary Conference on Web Accessibility (W4A)

  Commissioned by ANEC (the European consumer voice in standardisation), this study examined whether voluntary self-declaration or third-party certification of web accessibility conformance translates into actual compliance with WCAG 2.0. The researche…

  • WCAG compliance
  • accessibility evaluation
  • certification
  • conformance testing
  • e-government
  • European accessibility
  • accessibility policy

• Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole Reliance on Automated Tests

  Markel Vigo, Justin Brown, Vivienne Conway (2013) — Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (W4A)

  This paper provides rigorous empirical evidence on the limitations of automated web accessibility evaluation tools by benchmarking six state-of-the-art tools — AChecker, SortSite, Total Validator, TAW, Deque, and AMP — against expert manual evaluatio…

  • automated testing
  • accessibility evaluation
  • WCAG compliance
  • evaluation tools
  • benchmarking
  • false positives
  • conformance testing

• Evaluating Accessibility-in-Use

  Markel Vigo, Simon Harper (2013) — Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (W4A)

  This paper introduces a novel method for evaluating web accessibility that goes beyond traditional guideline conformance testing. The authors argue that standards like WCAG 2.0 capture only about 53% of accessibility problems experienced by users, mi…

  • accessibility evaluation
  • user interaction
  • behavioural analysis
  • screen readers
  • blind and low vision
  • automated testing
  • conformance testing
  • user experience

• A Task Assignment Strategy for Crowdsourcing-Based Web Accessibility Evaluation System

  Liangcheng Li, Can Wang, Shuyi Song, Zhi Yu, Fenqin Zhou, Jiajun Bu (2017) — Proceedings of the 14th International Web for All Conference (W4A)

  This paper addresses a practical challenge in scaling web accessibility evaluation: how to effectively assign manual evaluation tasks to volunteer crowdsource workers with varying levels of expertise. While automated tools can check many accessibilit…

  • web accessibility evaluation
  • crowdsourcing
  • machine learning
  • automated testing
  • conformance testing
  • task assignment