Ground
Works

The Green Light SONATA project originated with a hunch in the engineering domain, but could only materialize through true collaboration of researchers working across disciplines. The project tested the hypothesis that translating simulated traffic information into music could lead to musical resolution of persistent traffic congestion. Our team—consisting of civil engineers, a composer/performer/computer music researcher, and an ethnomusicologist—proceeded to construct a model of an intersection in which each direction of traffic flow was assigned a musical pitch. Hearing these pitches as cues, musicians could interact with the sonified traffic to allow vehicles to proceed through the intersection. The result was a musical “gamification” of traffic flow in which the goal was to minimize the vehicles’ idle time. The next stage of this project will include public demonstration and testing sessions, involving students and additional musicians, to assess the concept’s viability, refine our methods, and gather further data. The team’s multidisciplinary dialogue takes us on productive tangents translating between different domains of musical and technical expertise. Moving forward, we plan to test additional methods of data sonification, manipulating additional musical variables (including pulse and rhythm, sequence, scales, ornaments, and other musical elements) and scaling up to model multiple consecutive intersections.

Watching and Dreaming is a body of work that enables the viewer to peer inside the “mind” of a machine to observe its perceptions, mind wanderings, and dreams. This is not a metaphorical representation of dreams, nor a technical exercise in AI such as DeepDream [1] but the realization of a computational model of dreaming informed by cognitive neuroscience. This level of description avoids biases towards Jungian and Freudian psychology that assume dreaming is exclusively human. Dreams should not be considered independently of the perceptual capacities of the dreamer, and thus comparing this model to human perceptual abilities is problematic. For the audience, these artworks function as entry-points to consider the constructed nature of perceptions and the continuity of waking, mind wandering, and dreaming. For the artist, the artworks are sites of knowledge-making; it is through the making of artistic works that the model (computational formalization) and theory (argument that situates the model in empirical knowledge) are developed. The research underlying these artworks integrates knowledge in multiple disciplinary dimensions: (a) The computational modeling of dreaming processes (Zhang 2009; Treur 2011), (b) generative and media artworks engaging with the concept of memory and dreaming (Franco 2007; Dörfelt 2011), and (c) the conception of dreaming as imagination (Nir and Tononi 2010). In this text, Watching and Dreaming (2001: A Space Odyssey) (2014) serves as an exemplar of the Watching and Dreaming body of work. The machine attempts to learn and predict Stanley Kubrick’s film 2001: A Space Odyssey through the construction of its own subjective perception that is the basis of dreaming. “Mental” images generated during perception, mind wandering, and dreaming are subjective constructions bound to the peculiarities of the machine’s way of seeing. The body of work constitutes various manifestations of the cognitive model, not attempts to communicate the model’s mechanisms.