Harmonizing the Senses: Designing a Cross-Modal Interactive Art System to Enhance Older Adults' Affective Experiences

This CHI 2026 paper investigates how cross-modal correspondence and congruency across visual, auditory, and kinetic modalities shape older adults' affective experiences in digital art systems, and how an AI-infused drawing mode can scaffold those experiences. The authors frame the work against demographic context (the over-65 population is projected to exceed 1.5 billion by 2050) and against age-related declines in perception, cognition, and emotional regulation that can limit engagement with poorly calibrated multisensory stimulation (MSS). The study uses two linked experiments and a follow-on evaluation. Experiment I (N=35, ages 65-79) examines timbre-color associations with ten traditional Chinese instruments and the 37-color Berkeley BCP-37 palette, using PrEmo cards to measure emotional ratings. Participants were stratified by Montreal Cognitive Assessment-Basic (MoCA-BC) scores into low- and high-cognitive-ability groups. Experiment II (N=32) builds a TouchDesigner-based cross-modal ink-painting system on a Wacom tablet, mapping stroke speed to timbre rhythm and stroke vertical position to timbre pitch across five synchrony conditions (no correspondence, correspondence without congruency, speed-rhythm, position-pitch, combined). Facial expression data were captured with Noldus FaceReader 9. A third phase adds an AI-infused mode using Stream Diffusion, a VAE, and ControlNet to transform strokes into real-time Chinese ink-style artworks, evaluated against the non-AI condition. The work targets both empirical contributions on affective congruency and design guidelines for emotionally supportive MSS systems for older adults.

Experiment I confirmed an emotion-mediated link between timbre and color: brighter, sharper timbres predicted saturated, light, yellow-red color choices (PMCA_S/U beta=0.891, p<.01; R^2=.793), and joyful/hopeful/desirous timbres aligned with the same visual attributes. Experiment II found no significant main effect of synchrony condition on self-reported affect, but a significant t-test result showed lower-cognitive-ability participants reported higher pleasure than higher-ability peers under the correspondence-without-congruency (CNC) condition (M=0.20 vs 0.08, p=.006) — many described high-synchrony modes as 'overwhelming' or 'distracting'. The AI-infused evaluation (paired t-tests, N=32) produced the strongest results: AI mode significantly increased pleasure (mean diff=0.494, p<.001), surprise (0.713, p=.003), and reduced sadness (-0.213, p=.025). Facial-expression ANOVA showed AI significantly raised valence (eta_p^2=.311) and arousal (.234) and lowered sadness and anger. Crucially, AI compensated for cognitive ability: low-cognitive-ability participants, who showed a sharp decline in pleasure and valence and a rise in sadness without AI, matched or exceeded high-ability peers with AI support.

For accessibility practitioners working with older adults and people with cognitive decline, the paper offers concrete evidence that high-synchrony multisensory output — often promoted as richer or more immersive — can overwhelm users with reduced cognitive flexibility and sensory tolerance. Lower-complexity, predictable pairings should be the default, with optional AI scaffolding that lowers operational demands while preserving user agency (the authors call this 'graduated autonomy'). The finding that AI-infused creative tools closed the gap between cognitive-ability groups is relevant for care-home and rehabilitation settings where mixed-ability groups share the same equipment. Limitations to flag: the sample skewed toward digitally familiar, active older adults from one community center in Mandarin-speaking China, so findings may not transfer to users with severe sensory impairment, lower digital literacy, or rural contexts. The study was short-term and lab-based — novelty effects may inflate AI's apparent benefit — so longitudinal deployment in real settings remains an open question.