Despite the well-known benefits of visualization, much of the cognitive mechanisms and processes that give rise to these benefits are still not properly understood. On one hand, visual perception of individual objects in the visual scene is a well researched and understood topic. The psychology literature is full of articles on visual perception, pre-attentive processing, and visual working memory, with impressive experimental results. On the other hand, the higher-level cognitive operations a viewer employs to visually reason about data depicted in a visualization are less understood. These cognitive operations surely must involve interactions between perceptual systems, visual working memory, and perhaps deeper semantic knowledge. Yet, many questions remain on how these cognitive operations precisely work to bridge the gap between seeing and understanding. My research in this area attempts to shed a light on these questions.
ACT-R model of visual problem solving
In one project, I designed a cognitive model using the ACT-R cognitive architecture to study what Collin Ware terms Visual Problem Solving. The model predicts user performance time (in seconds) needed to answer simple questions about data depicted in a visualization. The model comparers two informationally equivalent visual representations (tree vs. icicle plot).
Understanding the interaction between visual and verbal metaphors from eye gaze behavior
In a second project, we replicated Ziemkiewicz et al's experiment on the interaction of visual and verbal metaphors. In addition to analyzing the accuracy and response time of subjects, we also tracked their eye gaze. This allowed us to look at how users approach a visualization task given different verbal metaphors in the task instructions. We replicated some of Ziemkiewicz et al's result. We also found significant differences in eye gaze behavior and strategy, when users are given verbal instructions that are either compatatible or incompatible with the visualization's underlying visual metaphor.
Our results suggest that users are able to predict the structure of a visualization better when prompted with compatible verbal instructions. This predictability manifest itself in better-targeted eye gaze patterns; users were able to quickly "jump" to the location containing the solution to the visual problem, given a verbal metaphor that is compatible with the visualization.