In dialogue with a learning machine: an exploration of music, interactive theatre and live performance through Artificial Intelligence

Lure is an artistic research project related to machine learning in theatre and music. We have explored a variety of machine learning techniques in combination with musical instruments, voice and narration. The artistic research transfers questions on the topics of human–machine interaction and artificial intelligence to the fields of theatre and music.

The narrative is based on the classic novel Parzival by Wolfram von Eschenbach. Parzival meets three elegant knights, decides to seek King Arthur, and continues a spiritual and physical search for the Grail, looking for the world outside the forest in which he was raised. In Lure we have used the character of Parzival, who is now developed inside a computer, with a similar intentionality. It wants to learn about you and the world and starts a conversation with you. It tries to express itself by playing with a musician. And it leaves its virtual confinement to explore the world by itself.

With our three prototypes we studied three different artistic and technological approaches. Develop empathy with the computational character by discovering how it learns. Experience its inner workings through theatrical expression. And be touched by its musical sensitivity.

We developed an open studio event where we invited an audience for a playful session with music, performance and collective discovery. The audience was asked to interact with our character Parzival. It was an interactive presentation of our research and the three prototypes as artistic steps and even though it is work-in-progress, it was presented in a theatrically staged online performance.

Core people involved in the project are Munich composer and theatre maker Mathis Nitschke, Amsterdam experience artists Luciano Pinna and Klasien van de Zandschulp, scientists from the Munich Volkswagen Group Machine-Learning Research Lab (argmax.ai) Patrick van der Smagt, Djalel Benbouzid and Nutan Chen and Munich dramaturg Elsa Büsing.

 

Full Performance Video

Originally meant to happen in the physical space of Munich Pathos theatre, this project became also a prototype for a theatrical staging in digital space, due to Corona. As an audience member, imagine yourself being one of the participants in this Zoom-like meeting. You enter the space with your webcam using the same custom built Augmented Reality filter which you see on all the faces in the video.

We invite you to darken your room to get the full experience.

Chapters

00:35 PARZIVAL runs through the forest

04:10 PROTOTYPE 1: Singing with the Birds.
Parzival in musical interaction with a singer. An artistic idea of a musical dialogue between machine and performer. The user interface shows a learning process that could happen between performer and musician.

09:43 PARZIVAL and Herzeloide

12:43 PROTOTYPE 2: Jamming with the Knights.
Parzival, the machine, in playful interaction with a drummer. Simultaneously making visibly understandable what happens inside the machine.

21:00 PARZIVAL meets the knights

22:50 PROTOTYPE 3: A Chat with you, Angels!
Parzival is curious to chat with you at lure.bsqd.me

30:33 PROTOTYPE 4: What does it look like where the forest ends?
Parzival uses your answers to collect a corpus of images which it uses to make itself a picture of the world. Parzival leaves the space.

 

Q&A Session

directly after the Lure Open Studio Performance on 13 November 2020. With (left to right, top to bottom) Mathis Nitschke, Elsa Büsing, Luciano Pinna, Katharina Dobner, Martina Koppelstetter, Nutan Chen, Klasien van de Zandschulp, Patrick van der Smagt, Mara Widmann, Matthias Lachenmayer and Djalel Benbouzid

 

Review

Rita Argauer wrote a wonderful review in Süddeutsche Zeitung: “the brief moment when computer and human being came together artistically is full of horror and beauty, ambivalence and theatricality. To create something like this through a miserable computer screen is a great achievement.”

Matthias Lachenmayer performing with the FMVAE model created by Argmax.ai

Invitation text

A curious and creative character awakens. Developed inside a computer it yearns to learn. You meet it in our performance. It wants to learn about you and starts a conversation. It tries to express itself by playing with a musician. And it leaves that space to explore the world by itself.

In our open studio we invite you to a playful session with music, performance and collective discovery. You’ll be able to interact with our character Parzival, a machine-learning model created after the mediaeval adventurer.

With our prototypes we study three different artistic and technological approaches. Develop empathy with the computational character by discovering how it learns. Experience its inner workings through theatrical expression. And be touched by its musical sensitivity.

The artistic research project “Lure” transfers questions on the topics of human–machine interaction and artificial intelligence to the fields of theatre and music. We will open our studio and show our first technical prototypes and artistic steps as work-in-progress, in a theatrically staged online meeting.

Katharina Dobner filming the Lure fire

Credits

“Lure” is a project of Munich composer and theatre maker Mathis Nitschke, Amsterdam interactive artists Luciano Pinna and Klasien van de Zandschulp, scientists from the Munich Volkswagen Group Machine-Learning Research Lab (argmax.ai) Patrick van der Smagt, Djalel Benbouzid, Nutan Chen and Munich dramaturg Elsa Büsing.

with
Mara Widmann: Narrator
Martina Koppelstetter: Singer
Matthias Lachenmayer: Drummer
Katharina Dobner: Mask and fire design
Dominik Theobald: Digital Tree

Musical contributions by gamut inc, Jörg Hüttner, Martina Koppelstetter, Matthias Lachenmayer, Mathis Nitschke, Alexander Schubert

Parzival’s learning machine is based on the flat latent-variable models explained in https://argmax.ai/blog/metric/.

Besides our custom development tools we use these projects: Google Magenta, Tensorflow Pix2pix, OpenAI GPT-2, RunwayML, AIVA, Botsquad, Lens Studio, Av3ry

“Lure” is funded and supported by the Dutch Creative Industries Fund NL, the German Fonds Darstellende Künste, and the Volkswagen Group Machine Learning Research Lab.

Thanks to Pathos Theater, Mario Klingemann, Memo Akten, Vladimir Viro, Arjan Scherpenisse, Botond Cseke, Milo Schagen, Michael Schwarzfischer, Markus Krause, Tankred Dorst and Wolfram von Eschenbach.

 

Ending Sequence with a pix2pix model trained on an image corpus collected on the basis of audience answers in the chatbot.