Close Reading as a Method for Evaluating Visualizations
In the course of iterating on and measuring the success of visualizations created under the aegis of SculptingVis, we realized that traditional methods of evaluation could not fully account for our goals and research questions. While task-based forms of evaluation can tell us important information about some of the ways that users interact with a visualization, they cannot tell us whether or not visualizations are effective in their abilities to elicit qualities like depth of thought or pleasure that are difficult to measure quantitatively.
To formulate new modes of evaluation that complement the aims of SculptingVis, we turned to the humanities, a discipline which for centuries has honed methods of analyzing, interpreting, and evaluating works of art and cultural productions. Following the lead of humanities scholars who have analyzed data visualizations as cultural objects, we evaluate visualizations that utilize arts and design principles by leading non-experts in close reading, a widely-employed practice across the humanities that activates viewers’ analytical capacities, asking them to break down features of the visualization, contextualize these features, determine which features are most significant, and from these qualities, to arrive at a conclusion about the meaning of the visualization.
This method offers a way of evaluating how various features of a visualization come together to create meaning about their subject and to stimulate analytical thought in a viewer. It aims to not only evaluate what a visualization leaves a viewer with (specific information, an emotion, another question) but how it does so through specific features. It provides an alternative method to current modes of evaluation in that it lends itself to richer, more open-ended insights from participants than traditional surveys, but is also a more replicable, less time- and resource-consuming method than one-on-one interviews. As this method is derived from a pedagogical tool, we also expect that it can inform practices for teaching visualizations.
A forthcoming Visualization Viewpoints discussion in IEEE CG&A describes our motivation for using close reading as an evaluation method, as well as our first study procedure and its results in detail. We’ve made a visualization close reading worksheet template available for download for those who would like to replicate this evaluation method or for teachers who want to use it as a tool for teaching students how to analyze visualizations.