Fractal Choreography using Motion Capture and Haptic Sensors

In this article, we present a novel way of choreographing movement with fractal feedback using motion caption and haptic sensors. The practitioners receive a visual, sonic, and kinaesthetic signal about their current dynamic state, which they can interpret to change their movement pattern or to maintain it further.

As a result, the movement gets more diverse and unpredictable. Fractal feedback encourages practitioners to change their movement patterns and try something new. The focus shifts from aesthetics towards dynamics. The figure of the choreographer fades away as the open-source code is used to create choreography. The movement itself becomes more fractal in essence: self-similar across different scales, a combination of slow and fast oscillations, pauses, long stretches, and breaks in between.

Below we present a short excerpt of the choreography made using fractal feedback as well as a detailed explanation of the methodology used.

We used this approach to create choreography for the Gwangju Biennale 2021 opening, so we also add some footage of the rehearsals to share the process.

 

Data as a Mirror

If you are a movement practitioner you’ve probably used some sort of feedback in your practice. It may be a mirror, a video camera, or a fitness tracker. The feedback that they provide is pretty straight-forward: you see your movement as it is, in real-time, so you can learn how to move in a certain way, replicate a dance phrase, reach an exercise goal, or achieve a certain form or aesthetics.

EightOS tripod haptic feedback sensor

What if the mirror or the camera lens were distorted? What if the fitness tracker analyzed the pattern of your movement and provided additional information, not usually available to your senses? The feedback you get becomes more interesting. It’s not so direct anymore. You start to see what you don’t normally see and this makes you move in a different way. The patterns break, the new sensations emerge. It’s not so much about concrete shapes and results, but more about the physical and emotional states. Movement as creative exploration.

This is the idea behind the EightOS Fractal Choreography framework that we present in this article. The main idea of this approach is to provide feedback about the fractal signature of the movement dynamics, which can detect four distinct states.

The computer system detecting the current dynamical state: repetitive, regular, fractal, or complex. Each symbol represents a state and a corresponding alpha exponent is sent via ODP to Usine and Orca to generate sound and haptic feedback via the EightOS sensor. photo: EightOS at Gwangju Biennale, 2021

The practitioner can then respond to this feedback and focus on maintaining a certain state or shifting between them. The objective is not anymore about fitting a mold but creating certain dynamics, sculpting in time with one’s own body, and exploring the implications of this approach on one’s practice. The resulting dynamics may also affect physical performance. It has been shown that “healthy” movement dynamics that is adaptive and resilient is also fractal in essence: it is self-similar across different scales. So there are both aesthetic and practical consequences arising from this approach.

Moving to the fractal feedback impulse

 

How Fractal Feedback Works

1. xSens Dot motion capture sensors are attached to several parts of the body or to each practitioner in a group. They will feed the data about the movement acceleration (for the body part they are attached to: for instance, the waist or the hand).

The xSens motion capture sensor is attached to the hand, sending the acceleration signal to the EightOS fractal control software, which then provides movement feedback to the dancer

2. Using a combination of xSens server software and our own EightOS Fractal Analysis server app (running on a computer or Raspberry Pi for portability) we activate motion capture sensors and set up the data stream (via Bluetooth), which is then analyzed in the real-time using Detrended Fluctuation Analysis (DFA). The result is the alpha coefficient, which can be interpreted as 4 different states: repetitive (random), regular (correlated), fractal, and complex.

The projection screen and the workstation processing the signals (video still)

3. The alpha coefficient is then projected to provide direct visual feedback to the practitioner (you can see it in the screenshots below). They can use this feedback to encourage certain movement dynamics or to change what they are doing.

4. The alpha coefficient signal and the state is also sent via OSC (a network transfer protocol) to Usine software, which activates the signal in the EightOS tripod haptic feedback sensor via wi-fi. The EightOS sensor will then provide physical feedback to the practitioner, in addition to the visual, so that they can modify or maintain their movement pattern.

The EightOS fractal tripod sensor

5. The OSC signal can also be used to generate sound, which is done using a combination of Usine and Orca software. Every state is represented with a certain sonic soundscape and can be used as additional feedback and inspiration by the practitioners, providing a musical choreographic score.

 

 

Implications and Effects on the Practice

We’ve experimented with this approach during the process of creating the choreography for the opening of the Gwangju Biennale 2021. This was a great opportunity to test this approach and to gather some feedback from dancers about their experience. Here are some observations we’ve had so far and you can also watch a short video with the description of the approach below.

• It is a great way to change your own movement patterns and to try something new. You start to move in a different way, exploring the different ways of moving in new ways, liberated from the limitations imposed by your aesthetic preferences and habits. You are not so much concerned with the form, it is more about the dynamics.

EightOS rehearsal at Gwangju Biennale, 2021

• The focus shifts from aesthetics to adhering to certain principles. This also changes the approach to movement where it is not about replicating a certain shape or performing a set of phrases, but, rather, creating your own dynamic sculpture in time, using your own approach and physicality.

The figure of the choreographer fades away. Every mover becomes their own choreographer guided by the algorithm. One can play with the algorithm by changing the response to the feedback that it provides. For example, you can focus on maintaining the fractal dynamics or the repetitive one. You can also focus on variability and shifting between the states. You are your own choreographer using the sensors to observe yourself.

EightOS rehearsal at Gwangju Biennale

• It is easier to believe in the practice and to be confident of the movement. The principles are simple, the feedback is precise, based on the scientific data and predefined logic. So it is easy to follow the guidance, which emerges from the interpretation of your own data: whether you choose to track your alpha score, observe the symbols, or follow the soundtrack produced by the dynamics states you create.

• The technical setup we use also provides a concrete way to link the physical movement, data, the digital and technical component, the visual, and the sonic registers. The body can influence the visual aspect, which, in turn, will have an effect on the sound, which affects the movement again, creating an ongoing feedback loop across the different modalities.

Moving to the fractal feedback, photo: EightOS at Gwangju Biennale, 2021

• Unlike traditional feedback mechanisms that are focused on some sort of ideal (e.g. a movement performed in a certain way, a dance phrase, or a physical exercise), here the objective is not a fixed form but, rather, a dynamical state, which changes one’s approach to movement and pushes one to explore physicality from a completely different perspective.

Special thanks to all the dancers: Lyon Eun Kwon, Yi Yeji, Jonghyeon Lee, Choi Sengmuin, Ji Hye Chung, Yunju Lee

And to the Biennale and curatorial team: Jinyoung Shin, Davide Quadrio, Natasha Ginwala, Defne Ayas 

EightOS tripod haptic feedback sensors by: Julien Thomas and Kirikoo Des

Choreography, research and visuals by: Dmitry Paranyushkin and Kirikoo Des

Software by: Dmitry Paranyushkin, xSens, Julien Thomas and Kirikoo Des, Orca (HundredRabbits), Usine.

Read more about the science behind this approach in the Nodus Labs article on fractal variability.