ASD is a life-long condition characterized by impairments in social interaction and communication. Individuals with ASD also have restrictive, repetitive behaviors (CDC, 2014), and anxiety affects approximately 80% of people on the autism spectrum (Smith & White, 2020). Yet, while understanding social patterns produces challenges, many on the spectrum possess a high aptitude for perceiving architectural, spatial, mathematical, musical, and motion patterns (Crespi, 2016). Autism advocate Temple Grandin labeled those who create patterns to establish order, meaning, familiarity, and calm -- “pattern thinkers” (Grandin, 2013). Choreography often focuses on manipulating patterns to music, and the penchant for patterning has been shown to be soothing for many people with ASD (Scharoun et al, 2014). Could an immersive environment that produces opportunities for creating choreographic patterns have a palliative effect on anxiety?
Despite the strengths and abilities of people with ASD, most ASD research focuses on early intervention with the goal of diminishing impairments, rather than acknowledging that it is a syndrome with differential strengths and abilities (Hendricks, 2010). Instead, we took a strength-based approach, using game design practices and ideas that have been refined over several decades to improve quality of life, educate, encourage positive behavior, and facilitate healthcare needs (Ratan and Ritterfeld, 2009).
Initially proposed as a tablet-based app, the project that would become Choreografish pivoted to VR early in the process, following the literature on virtual reality and autism (Hong & Mi-Sung Kim, 2020). This allowed us to match the research interests of our collaborators with ASD and to investigate the arts-and-health potential of VR’s full perceptual immersion and capacity for producing wonder.
Integrating Choreographic Thinking and Virtual Reality
Choreography is a diverse artform. However, we found that constraining the definition of choreography helped translate general aspects of its Western concert practice to our game designers, computer engineers, and young adult collaborators with ASD who had never choreographed. We began thinking of choreography as a form of pattern thinking that emphasizes the interplay of moving objects and their coordination to music. The concept of “choreographic thinking” first came to our attention through the Synchronous Objects dance-and-data-visualization collaboration between former Frankfurt Ballett choreographer William Forsythe, Ohio State University (OSU) Professor of Dance Norah Zuniga Shaw, and computer visualization specialists at OSU. Their work suggests that choreographic thinking is a variant on the human impulse for organization, where intentionally designed relationships between moving objects are crafted to effect perceptions, sensations, actions, ideas, and emotions. Importantly, Forsythe and Zuniga Shaw’s work seems to frame choreographic thinking as a cognitive mode that can be generalized beyond dance—the social aspects of which can be a barrier to entry for people with autism spectrum disorder—prompting us to adopt their expansive definition of “choreography” from among many other possible conceptions of that term.
Our first iteration on a user interface involved a virtual pedestal-based control system that translated select choreographic variables into mechanically analogous digital controls like sliders and buttons. For simplicity, we delimited the choreographic variables for a school of fish to speed and scale (expansion and contraction of the school). These two variables, tempo and scale, enable the user to effect changes within a limited field, and potentially create a sense of music-based change. We simplified play in order to 1) reduce the risk of overstimulation and 2) translate the unfamiliar and mutable concept of “choreographic thinking” into clearly defined development goals for the game designers and engineers.
Choreografish players use the HTC Vive, a commercially available virtual reality headset to explore and interact with an expansive underwater world. The game allows users to upload any music of their choice and begin synchronizing the swim patterns of schools of fish. Users can record the swim patterns they create and play them back for review. The user can “teleport” to various spaces in the environment (including onto the back of a whale) to witness their creation from multiple vantage points. We drew design and engineering cues from a number of different games, especially the VR painting game Google Tiltbrush. Inspiration for movement and teleportation came from The Lab, and BOID behaviors were included to animate the schools of fish.
RESEARCH And Ongoing Questions
University of Utah Institutional Review Board-approved focus groups and interviews have been essential to understanding the user experience of Choreografish. The autism researchers on our team led the process, acquired consent, coordinated demos and feedback sessions, and coded data. Game designers and engineers led the demos and focus groups, and transformed the raw data from interviews about the user experience (UX) and user interface (UI) into action items to develop the software. Dance provided the theoretical framework of choreographic thinking that helped team members describe, interpret, contextualize, and evaluate the effectiveness of the UI and UX.
This iterative, heuristic feedback loop produced insights into the intuitive and emotional states that emerged through the VR experience and helped us make game-development decisions that were appropriate to the feedback gained from our research population partners. It has also led to new questions and challenges.
While focus group data confirmed the calming nature of the underwater environment, the same data also suggested the ineffectiveness of the design for the choreographic controls. The pedestal reflects an attempt to translate basic choreographic variables into a logical form, but the design constrains the role of the body, limiting its improvisational, idiosyncratic, and intuitive potential for producing change in the environment. Although working the sliders together in a rhythmic fashion can produce the illusion of a dancing school of fish, we came to see that the restrictive movements of the sliders interfered with freer, more fully embodied rhythmic responses. The restrictiveness of our design unexpectedly led to a longer learning curve and necessitated the development of a virtual tutorial to explain the range of available creative options. Furthermore, the constraining effect that we observed our controls had on the body was compounded by the fact that virtual reality is somewhat disembodying, rendering a person’s body invisible to themselves and making them reliant on proprioception and imagined tactility. We have not yet cracked the code on creating intuitive controls to facilitate “flow” in the body.
ConCLUSION AND Future Plans
Since high degrees of complexity cause anxiety in the lives of young adults with ASD, we are curious to see if Choreografish affects anxiety levels by enabling people to exercise order over complexity. Choreografish provides a model for co-design with youth with autism. It also illustrates the potential for interdisciplinary collaborations, in this case artists, social scientists, and game designers, to develop novel and innovative software solutions to address complex health concerns. The research team has begun to conduct trials to measure the impact of the game on anxiety for youth with ASD. Initial results are promising, showing reduced anxiety in a yet-to-be published feasibility study, but more research must be done before a claim of efficacy can be made. The team strongly believes that embodied choreographic thinking in virtual reality may have potential for leveraging pattern thinking in a non-deterministic, arts-based way to help young adults with ASD self-soothe through an immersive, creative engagement with technology.
- Brandt, E. (2006). Designing Exploratory Design Games: A Framework for Participation in Participatory Design, Proceedings of the Ninth Conference on Participatory Design: Expanding Boundaries in Design (Vol. 1, pp. 57-66). ACM.
- Caron, M., Mottron, L., Rainville, C., & Chouinard, S. (2004). Do high functioning persons with autism present superior spatial abilities? Neuropsychologia, 42(4), 467-481.
- Center for Disease Control and Prevention. (2014) Prevalence of Autism Spectrum Disorder Among
- Children Aged 8 Years — Autism and Developmental Disabilities Monitoring Network, 11 Sites, United States, 2010. https://www.cdc.gov/mmwr/pdf/ss/ss6302.pdf
- Crespi, B. (2016). Autism as a disorder of high intelligence. Frontiers in Neuroscience, 10 (300). https://doi.org/10.3389/fnins.2016.00300
- Forsythe, W., Palazzi, M. & Zuniga Shaw, N. (2009) Synchronous Objects. https://synchronousobjects.osu.edu/
- Grandin, T. (2013). The autistic brain: Thinking across the spectrum. Boston, MA: Houghton Mifflin Harcourt.
- Hendricks, D. (2010). Employment and adults with autism spectrum disorders: Challenges and strategies for success. Journal of Vocational Rehabilitation, 2, 125-134.
- Hong, S., & Mi-Sung Kim, H. (2020). Virtual Reality-Based Intervention for Developmental Functioning in ASD: A Meta-Analysis. The Korean Journal of Developmental Psychology, 33(1), 85-100.
- Nicely, M. (2014, March 1). On Choreographic Thinking. Retrieved July 11, 2018, from https://dancersgroup.org/2014/03/on-choreographic-thinking/
- Ratan, R. & Ritterfeld, U. (2009). Classifying serious games. Serious games: Mechanisms and effects (pp. 10-24). New York, NY: Routledge.
- Scharoun, S. M., Reinders, N. J., Bryden, P. J., & Fletcher, P. C. (2014). Dance/Movement Therapy as an Intervention for Children with Autism Spectrum Disorders. American Journal of Dance Therapy, 36(2), 209-228.
- Smith, I., & White, S. (2020). Socio-emotional determinants of depressive symptoms in
- adolescents and adults with autism spectrum disorder: A systematic review. Autism, 1-16. https://doi.org/10.1177/1362361320908101.
- Spier, S. (2011). William Forsythe and the Practice of Choreography: It Starts From Any Point. Routledge.
- Valerie, S., Ventura, M., & Fengfeng, K. (2015). The Power of Play: The Effects of Portal 2 and Lumosity on Cognitive and Noncognitive Skills. Computers & Education, 80, 58-67.