Close Up Research: Multispecies Soundscapes

In November, AIL will host the next Laser Talk* Sonophagia – Eating Sound. On this occasion we sat down with Klaus Spiess, who is part of the team behind the Laser Talk project, and his colleagues Ulla Rauter and Emanuel Gollob, to give us more insight into their work and research. Elisabeth Falkensteiner (Head of AIL) was leading the conversation.

Klaus Spiess, Ulla Rauter, and Emanuel Gollob collaborate by integrating their expertise in medicine, media art, and artistic AI to investigate the interplay between human vocalization, the oral microbiome, and artificial intelligence. Spiess brings his background in medicine, anthropology and performance to explore the conceptual frameworks that challenge traditional human-centered communication, particularly focusing on the microbiome’s role in language and identity. Rauter, a media artist and researcher in transmedia arts, leads the investigation into how specific sounds and vocal expressions can influence microbial communities, creating experimental soundscapes. Gollob, working in creative robotics, develops and implements Deep Reinforcement Learning (DRL) models to enable real-time interaction between AI, human voices, and microbial behavior, emphasizing a dynamic cross-species dialogue. Together, they create interactive installations that blend science, art, and speculative design, inviting audiences to engage with and contribute to these evolving multispecies soundscapes.

This collaboration aims to explore the interactions between the human voice, artificial intelligence, and the oral microbiome, creating a multispecies soundscape where human and microbial agents collaboratively shape auditory experiences. By examining how different sounds and linguistic expressions impact the microbiome, they highlight the co-authorship between humans and non-human entities in creating vocal and phonetic outputs. Through installations like Chronolalia and Sonophagia, audiences engage in a spatial experience where their voices interact with AI and microbial cultures, fostering a sense of shared agency across species and technologies. The project seeks to rethink the human-centered notion of language, authorship, and identity, dissolving traditional boundaries between human and non-human contributions. Ultimately, it combines art and scientific inquiry to propose a posthuman commons.

Since 2023, Paul Gründorfer, an electronic composer experienced in transdisciplinarity, Mehrta Shirzadian, a PhD in molecular biology and current student in the Art & Science department at the Angewandte, and Jürgen Ropp, a PhD in Interface Culture at the University of Arts in Linz, have contributed their expertise to the project.

Klaus, you are a trained internist and psychoanalyst and you have spent the last ten years fostering collaborative partnerships between medicine, art and performance. What fuels your interest in language and/or sign systems, especially with regard to your medical/psychoanalytical background?

[KS] I have a personal history of speech impediments and have therefore always regarded speech acts as existing between a loss of control and desire to create. I used to experience psychosomatic complaints – the very same issues I would later treat in my patients. These complaints are characterized by an inability to translate emotions such as anger, grief, or anxiety into coherent pre-linguistic or linguistic symbols, manifesting instead as physical symptoms. In contrast to this inability, the eminent child psychoanalyst Donald Winnicott demonstrated how a (lonely) child can use its imagination to animate inanimate objects, from teddy bears to individual sounds, and thereby console itself. He referred to this process as ‘primary creativity’.

This conflation of both the inability and ability to animate inanimate (sound) objects led me to question the creative relationship between nature and culture in terms of signs and symbols. I was less interested in Peirce’s semiotics or Saussure’s notion of arbitrariness, and more drawn to Nabokov, who emphasized the mutual entanglement of natural and cultural symbols. Nabokov was fascinated by the fact that a butterfly displays such aesthetic forms as eyes on its wings, which serve no functional purpose other than the aesthetic. Elsewhere, Michel de Certeau referred to this deception (mimicry) as a pure, functionless luxury afforded by nature.

I thus began focusing on theories related to the collapse of signification, which refers to situations where specific conditions cause the conflation of sign and signifier. For example, as hunger intensifies to an extreme degree, the word for a particular food can come to represent the experience of hunger itself, until the two become indistinguishable. Psychoanalyst Dana Amir demonstrated this through recollections of life in concentration camps. In her book Poetics of DNA, linguist Judith Roof discusses the collapse of signifiers in relation to DNA, exploring its dangerous potential for commercialization. Margaret Atwood also engages with this idea in her dystopian novel Oryx and Crake, where genetically modified humans can survive on feces but lose the ability to create linguistic symbols, becoming capable only of naming things they can physically touch.

The climate crisis has also sparked my curiosity about ‘semiopollution’, which refers to pollution through pure signs. Although little research has been conducted in this field, this form of pollution encompasses a range of phenomena, from the acceleration of speech to the pollution caused by social media.

Ulla and Emanuel, your interdisciplinary artistic research project focuses on the interaction between the human voice and the oral microbiome. What do you hope to find out?

[UR] In 2022, while I was working on the project, we started to establish, question and investigate the interaction between the human voice and the oral microbiome on an artistic-speculative basis. Our primary focus was on the tonal qualities of the human voice and its linguistic transformation processes: How can different sounds, words, verbal and phonetic expressions affect the microbiome?

Based on multiple studies that illustrate the real effects of acoustic vibrations on yeast and other microorganisms, we established this relationship with the human voice. Through speech acts, the voice creates a variety of noises and sounds. From the hissing sibilants to the high overtones of vocals and diphthongs, there is a lot of sound material to affect the mouth's microbes in different ways.

What is the specific role of AI?

[EG] If you think about human speech as the product of a collaborative process between human agents and the oral microbiome, you start to wonder which other co-authors might be involved in our modes of verbalization. The role of technology in all its different forms should therefore not be overlooked. AI voice generators or speech-to-text transcription software illustrate the back-and-forth between AI and human speech, even if there are temporal or spatial lags compared to our current oral articulation techniques. Similarly, the reproductive cycle of our microbial cell cultures takes place on a different temporal scale, about 3 to 4 hours.

For this reason, we specifically decided to use Deep Reinforcement Learning (DRL) for its performative qualities and algorithmic open-endedness instead of other, more common AI speech generators. A Deep Reinforcement Learning (DRL) network, which has been trained on our lab data, is able to predict the microbial reproductive potential of different sounds and formulate performative responses to these projections and to current data from the microbial sensors – and thus mediates between microbial and human timescales.

With Chronolalia, a performance installation we presented at Ars Electronica in 2023, we sought to translate the ongoing process of oral co-authorship into an open-ended spatial experience, allowing visitors, the oral microbiome, and AI to become co-authors in the synthetic language of the space. Through this process, a visitor's language input, microbial reproductive needs, and the DRL's pattern preferences combine to create dynamic verbal expressions.

We technically constructed a feedback cycle that is both auditory and vibratory–tactile: Visitors are encouraged to mimic the synthetic voice articulator while standing on a whole-body vibration plate, providing more data for the continuous learning loop. In doing so, they explore new sounds with altered voices that may be beneficial for the reproduction of the microbiota growing in a bioreactor onsite. This echolalic speech investigates a potential space of unknown phonemes intersecting with the oral microbiota's needs, resulting in a polyphonic, vibrating, hybrid chorus of cross-species intelligence that operates in both microbial and vocal interests. When a visitor hears their voice and vibration distorted and alienated due to the microbiota's temporal growth, they perceive and understand the influence of the microbiota on their voice. In this moment, they experience themselves as a relational subject, whose singularity dissolves into a diffuse resonance or sound event. This experience addresses a polytemporal community with subjective components distributed among all actors: the speaker, former speakers, and the community of oral microbes with reproductive interests.

Here, AI approaches a practice of animism, which is increasingly relevant, as pattern recognition in unknown forms has often been regarded as a form of new animism. Michel de Certeau, elaborates on the animistic qualities of glossolalia, which informed another focus of our installation: the glossolalia-like multiple repetitions of sounds required to induce a change in cell structure.

Klaus, you also organize LASER Vienna Talks, co-hosted by Angewandte’s department of Media Theory, which focus on human sign systems as animate materials (signoids) and signifiers in terms of language, patterns or algorithms. You hosted four of these events at AIL. At the last event in November, ‘Making Souls – Making Bodies: Amerindian Cosmologies, AI, and the Microbiome’, you seemed keen to dismantle Western epistemic systems of knowledge production. What can we learn from indigenous ontologies or cosmologies about non-human intelligence, such as microbiomes and artificial intelligence? What specific challenges does the microbiome pose? What is so fascinating about them?

[KS] For a long time, I have incorporated ethnographic and medical-anthropological approaches into my research practice because I felt it was necessary to move beyond the hierarchical Western understanding of the nature-culture bind through an indigenous reading of this relationship. While our working group's installations are still largely based on Western thought—principles of cause and effect, scientific understandings of nature, linear time, and the great divide between human and non-human—I find it fascinating how Brazilian anthropologist Eduardo Viveiros de Castro challenges this system in his descriptions of Amerindian culture. The people of Amerindia regard animals as reincarnated humans and see humans as sharing a culture with animals, but not their bodies. In contrast, Western natural science tells us that humans share the same physiological body with animals but lack their culture.

This leads to an entirely new (ethical) understanding of our world when we consider the implications of killing animals that were once humans. This is especially true in the case of the microbiome, as its practical scientific achievements dissolve any clear boundaries between human and non-human. The microbiome attests to the fact that non-humans are part of us and outnumber our own human cells. Since indigenous peoples learn to alter their vocal melodies, pitches, and tonal patterns for new songs through communication with 'outsiders,' such as birds or other non-humans, we can consider learning tonal sounds to communicate with our microbiomes. This strange conflation of Amerindian and natural scientific imagination holds for me great yet untapped potential for new inquiries in the fields of cultural theory, natural science, and the arts.

We aimed to access and showcase this potential in our latest installations at Ars Electronica 2024 and ISEA 24 in Melbourne, the latter of which was solely dedicated to the relationship between electronic arts and indigenous culture.

The performance installation Sonophagia highlights, in a more literal form, the exploitation of Amazonian microbiomes by modern industries as an echo of the 19th-century rubber trade. We contrast Amazonian speakers, whose extremely biodiverse microbiomes and endangered low-phonetic tonal languages stand in stark contrast to European speakers, who exhibit poor microbiome diversity and high-phonetic speech. In the performance, a tenor representing Caruso from Werner Herzog’s Fitzcarraldo appears to float in a double-walled cylinder of his own salivary microbiome while attempting to sing Massenet’s dream aria. However, the aria is dramatically altered by the diversity needs of his salivary microbiome.

His counterpart, a phonograph made from a local Amazonian newspaper that critiques environmental destruction, plays the voice of an endangered local speaker, asking Caruso to perform his Italian aria in their native language, which typically incorporates whistling, humming, and whispering. The tenor’s aria becomes a tonal composition dictated by the microbial growth needs and the tonal structure of the endangered language. Real-time sound data and laboratory data algorithmically inform the system, which evolves throughout the exhibition. By the end, a kreol aria is created that considers both endangered microbiomes and diverse languages.

Klaus Spiess is associate professor at the Centre for Public Health at the Medical University of Vienna where he is heading a working group on Art & Science. Alongside Ulla Rauter and Ruth Schnell (former professor and director of the department of Digital Arts, University of Applied Arts Vienna), he also has been working on the PEEK (AR 687) research project 'Semiotic Sympoiesis for the Posthuman Commons' together with the Angewandte. Media artist Ulla Rauter studied Transmedia Arts and is a lecturer at the department of Digital Arts at the University of Applied Arts Vienna. External research partner and media artist Emanuel Gollob studied Design Investigations at the University of Applied Arts Vienna and is currently university assistant and PhD candidate at the department of Creative Robotics at the University of Arts Linz. In addition to several other collaborators, Mehrta Shirzadian, a PhD in microbiology and a student in the Angewandte Art&Science department, also contributes.