With a background in multiple disciplines, Mathis Nitschke spends his time giving contemporary music theatre a new face. He created The Planets, an inter-galactic soundwalk in 3D audio, which you can experience in many parks throughout the world.

Mathis’ app is one of the shortlisted pieces in the Sound Walk September Awards 2022. Here, he and his collaborators talk about the context of their work.

Smartphones and headphones are ubiquitous. Using those devices, available to anyone, the Munich-based sound art and design studio Sofilab conducts experiments which seek to unlock unprecedented possibilities for novel and exciting musical experiences.

With the immersive audio app The Planets, artistically concieved by Gunter Pretzel and Mathis Nitschke, and technically realized by Michael Hartung (née Klier), you can experience the famous eponymous orchestral suite by Gustav Holst, played by the Munich Philharmonic Orchestra, as an interactive audio walk in a park near you, increasingly all over the world.

How this started

In 2015, as an opera composer, Mathis Nitschke experimented with the idea of creating a music theatrical piece along the Munich river Isar, Vergehen, which you experience by walking it. When the VR revival hit in 2017, he immediately started trying out different recording techniques to explore the question of whether it would be sonically interesting – and also musically satisfying – to provide a listener’s perspective from inside the orchestra.

The artistic co-creator of The Planets, Gunter Pretzel, already had this inside perspective of the orchestra: he has been a violist with the Munich Philharmonic for over 30 years. He describes this experience inside the orchestra as so exciting and powerful that he, and many of his colleagues, have difficulties feeling the same excitement when they take the standard listener’s perspective in the concert hall. That’s why, over the course of his career, he has developed a number of projects that allow listeners to get intimately close to the musicians.

After Pretzel tried out Nitschke’s soundwalk Vergehen, Pretzel approached Nitschke with the idea to use smartphones and headphones to create this perspective, not knowing that Nitschke was already experimenting with this. As a proof of concept, they created the app Inside MPhil in 2019, which allows one to walk through the imaginary orchestra, set on a meadow. With this, you can experience one movement out of Schumann’s Frühlingssymphony, in high spatial resolution. You enter the orchestra acoustically and hear the instruments as if you were standing next to them, just as the musicians hear themselves and their colleagues: the violins, the oboe, the timpani, or whatever you want to hear.

The idea of letting the listener explore the musical sound field inside an orchestra is not new. However, we think that many experiments focus too much on the technical aspects. For example, many projects choose to record one instrument at a time to eliminate crosstalk of the sound sources. While this leads to good separation and localization of the individual instruments and prevents phasing issues introduced later by the binaural HRTF filters, the musical result lacks interest. After all, an orchestra must play as a whole, in a hall, in front of an audience!

So, for us, it was clear that we had to use a live concert recording – and that we would have to deal with all the technical compromises. But the music itself should remain uncompromised at all costs. If the music doesn’t work, nothing works!

The Inside MPhil app, effectively a GPS-controlled soundwalk, showed, as a successful proof of concept, that it is indeed possible to create a satisfying binaural interactive listening experience based on a live concert recording. As a result, a full-length symphonic experience was the next logical step, taking advantage of the learnings.

A full symphonic experience

“The Planets”, Op. 32, written between 1914 and 1917 by Gustav Holst (1874-1934) is a perfect balancing act between popular and sophisticated art music and thus became a model for much of the later Hollywood film music. Moreover, it is a work that challenges and showcases the qualities of an orchestra, its soloists and its conductor. All this made “The Planets” the ideal vehicle for the next edition of Inside MPhil.

The original idea was to make the experience playful with an interactive spaceship flight to the planets, thus spanning the individual episodes, the landings on the planets and finally the original recordings. However, it soon turned out that the presence of the music was so powerful that it could not be incorporated into a game idea of any kind. But it also became increasingly clear that the music  was capable of carrying the experience by itself. The task now was to lead an unbiased listener into this music unobtrusively but with increasing suggestiveness. The format of a loosely joined narrative developed, in whose formulations the appearance and character of the planets, on the one hand, and those of the sounds and motifs of the music, on the other, were interwoven. An outer path was created, which led from the nearest planets to the most distant, and at the same time an inner path deeper and deeper into the music. The immediate real environment recedes more and more. And the reached final “distance” is at the same time the vastness of the universe as well as the immediate and mysterious nearness of the music in ourselves.

In order to make this process not too heavyweight, there are occasional refractions and double levels of imagination, like for example the talking spaceship, the passing moon of Jupiter, the pattering of the asteroids of the Saturn, or the dangerous seductive power of Venus.

The respective landings on the planets every time is an impressive event. These landings are prepared by a careful dramaturgy of elaborately designed sounds, which should guide the user and at the same time get him in the right mood for the coming planet. Once he has landed, the original music comes into its own all the more strongly and unfolds with the full force of its artistic energy.

Inventing a visual layer

Originally, we envisioned the smartphone screen to be black and the phone to be pocketable so that the focus could remain entirely on the sound and the environment.

However, even after intensive research and development efforts, we have not found a way to reliably use the compass when the phone is not held in the default position. We use the compass to calculate the direction of the sound sources to the listener. If the vector of the phone to the listener’s body is not known, the listener’s relationship to the 3D sound environment is broken and immersion breaks down.

So we were faced with the task of developing a visual layer that was attractive enough to hold the phone throughout the listening experience, and at the same time not so attractive that it distracted the listener’s attention away from the sound and the environment.

Together with visual artist and programmer Luciano Pinna, we have developed a mechanism that transforms the current ground map into a star map: Paths and trails become Milky Ways, pulsars fly around, stardust fills the dark space.

The technical details

The Planets is created in the 3D game engine Unity, with the help of the commercial MapNav and Dear Reality add-ons.

MapNav is a Geolocation Engine for Unity and a powerful tool to develop location-based apps and games, including 2D/3D object geolocation, GPS navigation, and online maps. It creates a flat terrain inside the virtual 3D space of the game, based on outside map data, and hooks the moving player to the GPS and compass input. The player moves between 3D sound sources and colliders. We heavily modified MapNav to our needs.

The sonic heart of the app is Dear Reality’s Spatial Sound algorithm. The synthetic binauralization locates sound sources in the headphones coming from all directions of the three-dimensional space around the head. (In other words, you can not only control whether you hear the sound from the right or left, but also from the front or the back or up or down.)

The Munich Philharmonic records all their performances anyway, in their standard setup using main and spot microphones. Usually, they don’t use spot microphones on the brass as those are loud enough in the main mics. The orchestra’s recording producer Johannes Müller was extremely cooperative and provided not only the necessary spots on the brass but also, with Nitschke, used some more experimental techniques, like a quad outrigger setup, even if, in the end, these didn’t make it into the final product.

In the first Inside MPhil app, we took all these spot microphones as 3D sound sources and distributed them geographically on the map, the meadow. As a walking listener, one then moves ‘through’ these spot microphones. In the end, the user’s smartphone plays back a total of 22 tracks in real-time, all processed through HRTF filters. 

A big challenge is to keep CPU cost under control. The Holst orchestra is double the size of the Schumann, and on top of that, we now talk about a full symphonic experience of 1-hour in length. 40 simultaneously playing tracks, individually spatialised, are not feasible at all, for even the most powerful smartphones.

After some experimentation, we came up with a downmix to an octagonal circle with 8 audio sources through which the listener then moves. This technique works sufficiently well in terms of spatial exploration possibilities and reduces CPU and memory costs to 13 simultaneous 3D audio sources, including atmospheric FX and voice elements.

A low-threshold offer with high artistic standards

The Planets is a soundwalk experience whose precise dramaturgical and technical design evokes a subtle but impressive artistic process. Here, music becomes an experience that cannot replace a concert hall but is on a par with it in terms of experience intensity.

The app represents the appropriate contribution of the Munich Philharmonic Orchestra, an institution of international high culture, to musical life outside the established spaces. And like the successful balancing act of this music which was mentioned above, this presentation is now also an example of the unification of a low-threshold offer with high artistic standards.

This text is the fourth in a series of the artists shortlisted for the Sound Walk September 2022 Awards talking about their work.