Choreografish: an arts-based, virtual reality, anxiety intervention for autism

Eric Handman, Roger Altizer, Cheryl Wright, and Greg Bayles



Choreografish is a participatory research project leveraging virtual reality, arts engagement, and design to collaborate with young adults with autism spectrum disorder (ASD). The research team was motivated by a combination of observations: that some people with ASD experience social anxiety and attendant difficulties accessing social art forms such as dance and choreography, and that some have a predilection for developing patterns as a way of exerting control and making meaning. University of Utah faculty in engineering, dance, and social science collaborated with young adults with ASD on a virtual reality (VR) prototype to explore if synchronizing motion patterns to music may actually play well to the advantages of some on the autism spectrum and lower the barrier to a creative arts experience. Might choreographing in virtual reality help some people with ASD to self-manage anxiety?

View Project Site

Ongoing

Since July 2016

Sites and Institutions

School of Dance, University of Utah
GApp Lab for Medical Games and Apps, University of Utah

Keywords

Choreography Virtual Reality Autism Anxiety

Disciplines

Dance Game Design Autism Research





CONTEXT

ASD is a life-long condition characterized by impairments in social interaction and communication. Individuals with ASD also have restrictive, repetitive behaviors (CDC, 2014), and anxiety affects approximately 80% of people on the autism spectrum (Smith & White, 2020). Yet, while understanding social patterns produces challenges, many on the spectrum possess a high aptitude for perceiving architectural, spatial, mathematical, musical, and motion patterns (Crespi, 2016). Autism advocate Temple Grandin labeled those who create patterns to establish order, meaning, familiarity, and calm -- “pattern thinkers” (Grandin, 2013). Choreography often focuses on manipulating patterns to music, and the penchant for patterning has been shown to be soothing for many people with ASD (Scharoun et al, 2014). Could an immersive environment that produces opportunities for creating choreographic patterns have a palliative effect on anxiety?

Despite the strengths and abilities of people with ASD, most ASD research focuses on early intervention with the goal of diminishing impairments, rather than acknowledging that it is a syndrome with differential strengths and abilities (Hendricks, 2010). Instead, we took a strength-based approach, using game design practices and ideas that have been refined over several decades to improve quality of life, educate, encourage positive behavior, and facilitate healthcare needs (Ratan and Ritterfeld, 2009).

Initially proposed as a tablet-based app, the project that would become Choreografish pivoted to VR early in the process, following the literature on virtual reality and autism (Hong & Mi-Sung Kim, 2020). This allowed us to match the research interests of our collaborators with ASD and to investigate the arts-and-health potential of VR’s full perceptual immersion and capacity for producing wonder. 

Integrating Choreographic Thinking and Virtual Reality

Choreography is a diverse artform. However, we found that constraining the definition of choreography helped translate general aspects of its Western concert practice to our game designers, computer engineers, and young adult collaborators with ASD who had never choreographed. We began thinking of choreography as a form of pattern thinking that emphasizes the interplay of moving objects and their coordination to music. The concept of “choreographic thinking” first came to our attention through the Synchronous Objects dance-and-data-visualization collaboration between former Frankfurt Ballett choreographer William Forsythe, Ohio State University (OSU) Professor of Dance Norah Zuniga Shaw, and computer visualization specialists at OSU. Their work suggests that choreographic thinking is a variant on the human impulse for organization, where intentionally designed relationships between moving objects are crafted to effect perceptions, sensations, actions, ideas, and emotions. Importantly, Forsythe and Zuniga Shaw’s work seems to frame choreographic thinking as a cognitive mode that can be generalized beyond dance—the social aspects of which can be a barrier to entry for people with autism spectrum disorder—prompting us to adopt their expansive definition of “choreography” from among many other possible conceptions of that term.

Our first iteration on a user interface involved a virtual pedestal-based control system that translated select choreographic variables into mechanically analogous digital controls like sliders and buttons. For simplicity, we delimited the choreographic variables for a school of fish to speed and scale (expansion and contraction of the school). These two variables, tempo and scale, enable the user to effect changes within a limited field, and potentially create a sense of music-based change. We simplified play in order to 1) reduce the risk of overstimulation and 2) translate the unfamiliar and mutable concept of “choreographic thinking” into clearly defined development goals for the game designers and engineers.

Choreografish players use the HTC Vive, a commercially available virtual reality headset to explore and interact with an expansive underwater world. The game allows users to upload any music of their choice and begin synchronizing the swim patterns of schools of fish. Users can record the swim patterns they create and play them back for review. The user can “teleport” to various spaces in the environment (including onto the back of a whale) to witness their creation from multiple vantage points. We drew design and engineering cues from a number of different games, especially the VR painting game Google Tiltbrush. Inspiration for movement and teleportation came from The Lab, and BOID behaviors were included to animate the schools of fish.

Choreografish team members used Unity 3D software to render the various objects in the environment.
A computer monitor displays a graphic model of a shark to be used in Choreografish.
Photo: Dat Nguyen. / Copyright
An engineer tests the HTC Vive equipment. This testing process enabled the engineers and designers to integrate feedback from the various collaborators; translating aesthetic and affective responses into new functions and improved performance.
A person wearing a virtual reality headset holds a controller at arm’s length.
Photo: Dat Nguyen. / Copyright

RESEARCH And Ongoing Questions

University of Utah Institutional Review Board-approved focus groups and interviews have been essential to understanding the user experience of Choreografish. The autism researchers on our team led the process, acquired consent, coordinated demos and feedback sessions, and coded data. Game designers and engineers led the demos and focus groups, and transformed the raw data from interviews about the user experience (UX) and user interface (UI) into action items to develop the software. Dance provided the theoretical framework of choreographic thinking that helped team members describe, interpret, contextualize, and evaluate the effectiveness of the UI and UX.

This iterative, heuristic feedback loop produced insights into the intuitive and emotional states that emerged through the VR experience and helped us make game-development decisions that were appropriate to the feedback gained from our research population partners. It has also led to new questions and challenges.

While focus group data confirmed the calming nature of the underwater environment, the same data also suggested the ineffectiveness of the design for the choreographic controls. The pedestal reflects an attempt to translate basic choreographic variables into a logical form, but the design constrains the role of the body, limiting its improvisational, idiosyncratic, and intuitive potential for producing change in the environment. Although working the sliders together in a rhythmic fashion can produce the illusion of a dancing school of fish, we came to see that the restrictive movements of the sliders interfered with freer, more fully embodied rhythmic responses. The restrictiveness of our design unexpectedly led to a longer learning curve and necessitated the development of a virtual tutorial to explain the range of available creative options. Furthermore, the constraining effect that we observed our controls had on the body was compounded by the fact that virtual reality is somewhat disembodying, rendering a person’s body invisible to themselves and making them reliant on proprioception and imagined tactility. We have not yet cracked the code on creating intuitive controls to facilitate “flow” in the body.

A Choreografish user experiments with manipulating virtual fish as well as their environment.
Video: Dat Nguyen.

ConCLUSION AND Future Plans

Since high degrees of complexity cause anxiety in the lives of young adults with ASD, we are curious to see if Choreografish affects anxiety levels by enabling people to exercise order over complexity. Choreografish provides a model for co-design with youth with autism. It also illustrates the potential for interdisciplinary collaborations, in this case artists, social scientists, and game designers, to develop novel and innovative software solutions to address complex health concerns. The research team has begun to conduct trials to measure the impact of the game on anxiety for youth with ASD. Initial results are promising, showing reduced anxiety in a yet-to-be published feasibility study, but more research must be done before a claim of efficacy can be made. The team strongly believes that embodied choreographic thinking in virtual reality may have potential for leveraging pattern thinking in a non-deterministic, arts-based way to help young adults with ASD self-soothe through an immersive, creative engagement with technology.  

This trailer for Choreografish, originally designed for promotional purposes, demonstrates the possibilities for users to synchronize the basic actions of a school of fish to any music of the user’s choice. Other functions include pausing, rewinding and replaying the record of changes the player made to the school of fish, and teleporting to various other locations within the environment to witness one’s creation from different vantage points.
(2019)

Supplemental Material

This poster depicts early results of a study into the efficacy of Choreografish for reducing anxiety in young people with ASD.

Bibliography

  • Brandt, E. (2006). Designing Exploratory Design Games: A Framework for Participation in Participatory Design, Proceedings of the Ninth Conference on Participatory Design: Expanding Boundaries in Design (Vol. 1, pp. 57-66). ACM.
  • Caron, M., Mottron, L., Rainville, C., & Chouinard, S. (2004). Do high functioning persons with autism present superior spatial abilities? Neuropsychologia, 42(4), 467-481.
  • Center for Disease Control and Prevention. (2014) Prevalence of Autism Spectrum Disorder Among
  • Children Aged 8 Years — Autism and Developmental Disabilities Monitoring Network, 11 Sites, United States, 2010. https://www.cdc.gov/mmwr/pdf/ss/ss6302.pdf
  • Crespi, B. (2016). Autism as a disorder of high intelligence. Frontiers in Neuroscience, 10 (300). https://doi.org/10.3389/fnins.2016.00300
  • Forsythe, W., Palazzi, M. & Zuniga Shaw, N. (2009) Synchronous Objects. https://synchronousobjects.osu.edu/
  • Grandin, T. (2013). The autistic brain: Thinking across the spectrum. Boston, MA: Houghton Mifflin Harcourt.
  • Hendricks, D. (2010). Employment and adults with autism spectrum disorders: Challenges and strategies for success. Journal of Vocational Rehabilitation, 2, 125-134.
  • Hong, S., & Mi-Sung Kim, H. (2020). Virtual Reality-Based Intervention for Developmental Functioning in ASD: A Meta-Analysis. The Korean Journal of Developmental Psychology, 33(1), 85-100.
  • Nicely, M. (2014, March 1). On Choreographic Thinking. Retrieved July 11, 2018, from https://dancersgroup.org/2014/03/on-choreographic-thinking/
  • Ratan, R. & Ritterfeld, U. (2009). Classifying serious games. Serious games: Mechanisms and effects (pp. 10-24). New York, NY: Routledge.
  • Scharoun, S. M., Reinders, N. J., Bryden, P. J., & Fletcher, P. C. (2014). Dance/Movement Therapy as an Intervention for Children with Autism Spectrum Disorders. American Journal of Dance Therapy, 36(2), 209-228.
  • Smith, I., & White, S. (2020). Socio-emotional determinants of depressive symptoms in
  • adolescents and adults with autism spectrum disorder: A systematic review. Autism, 1-16. https://doi.org/10.1177/1362361320908101.
  • Spier, S. (2011). William Forsythe and the Practice of Choreography: It Starts From Any Point. Routledge.
  • Valerie, S., Ventura, M., & Fengfeng, K. (2015). The Power of Play: The Effects of Portal 2 and Lumosity on Cognitive and Noncognitive Skills. Computers & Education, 80, 58-67.

x

References

1 https://insar.confex.com/insar/2019/webprogram/Paper29573.html
2 https://www.researchgate.net/publication/328545961_Choreografish_Co-designing_a_Choreography-based_Therapeutic_Virtual_Reality_System_with_Youth_Who_Have_Autism_Spectrum_Advantages
3 https://dl.acm.org/doi/10.1145/3270316.3271541
4 https://www.youtube.com/watch?v=QYLGCwKO2x8
5 https://lassonde.utah.edu/choreografish/

Acknowledgements

The University of Utah Vice President for Research, The Entertainment Arts and Engineering Program, GApp Lab for Medical Games and Apps, NeuroVersity of Utah, photography and video by Dat Nguyen.