As art forms continue to converge, audiovisual experiences are emerging as a dynamic and immersive medium, blending visual art, technology, and sound into captivating installations. These experiences push the boundaries of traditional disciplines, offering composers and sound artists new creative avenues. As sound becomes an integral part of the artistic narrative, composers take on multifaceted roles that extend beyond music. This panel, which took place at the BMIM x ADE 2024 last October, invited us to dive into the evolving world of audiovisual experiences and the creative possibilities they offer.
In this panel:
Portrait XO is an award-winning multimodal artist and researcher known for pioneering work with AI-generated audio. As co-creator with Dadabots, she earned the VUT Indie Awards (2021) and the Eurovision AI Song Contest (2020). Her debut AI audiovisual album “WIRE” (2022) critiques biases in AI and its impact on creativity. She founded SOUND OBSESSED, fostering a community for exploring human-machine collaboration.
Merel van Helsdingen (moderator) founded Nxt Museum in 2020 as the first museum in the Netherlands dedicated to New Media Art, bridging the digital and physical worlds.
The start of audiovisual experiences from Portrait XO’s POV
Portrait XO started music in classical piano and violin, but later transitioned to electronic music production. She experimented with technology, meeting data scientists and learning about AI and audio. Her work focuses on using AI to create music and audiovisual experiences, often incorporating her own vocals and songs as training data. ‘It’s been pretty DIY, especially the production side. I had my own fair share of ups and downs of trying stuff and then not really feeling like anything was calling to me. And then AI opened up an interesting path for me.’
I had my own fair share of ups and downs of trying stuff and then not really feeling like anything was calling to me. And then AI opened up an interesting path for me.
The artist discusses how she combines sound and visuals in her work, using tools like Google Colab and AI models to create immersive experiences. ‘One of the inspiring things about where we are now with AI is the many ways we can create visual works. Even my professional visual artist friends feel the same. You can ideate quickly, and I’ve always wanted visuals to support my music.’ She explains how when she wants to start a track but doesn’t have the sonic inspiration yet, AI challenges her to get inspired by creating a visual and taking it from there: ‘It became this holistic way of working with audiovisuals.’
Trends in audiovisual performances
Portrait XO shares an example of a project where she used AI to generate visuals based on people’s dreams and nightmares during a residency in Berlin. While this is a very cool, but maybe more specific example, Merel van Helsdingen explains how AI takes over massive places in society as well. Think of ABBA, the group that runs a hologram show at the moment. Or The Sphere in Las Vegas, where techno artist Anyma popped up with his Afterlife show on 52.000 square meters of screen. But how does Portrait XO look at this trend? Does she see this in the spaces she performs? ‘Sometimes, not all electronic music fits well in a club setting, even if it has dance music elements. Some electronic music, especially when paired with audiovisual experiences, can be more impactful in a theater or seated environment rather than a typical club atmosphere. Otherwise, it doesn’t resonate.’
Outside of club spaces, AI and music, or audiovisual pieces, can also be used for good. Portrait XO: ‘There’s a common misconception that AI will replace human creativity, but I believe the opposite is true: AI can enhance creative expression if we confront our fears and lean into the possibilities.’ This realization hit the independent artist when she attended the AI for Good UN summit a few years ago, where she discovered that many policymakers responsible for critical decisions lacked a basic understanding of AI. She says: ‘It made me realize the urgent need for convergence between art, music, science, and technology. Fortunately, new tools are emerging that empower artists to collaborate with AI in innovative ways. For example, NewTone AI and Datamind Audio offer plugins that let musicians create custom AI models, while audiovisual plugins like EboSuite from EboMan enable new forms of artistic expression.’
‘One project that profoundly impacted me involved a set of Max for Live devices called Data Sonification Tools, developed by my friend Noah Pred at Manifest Audio. These tools convert numerical data into sound, and my first experiment used a century’s worth of global temperature anomaly data to create an audiovisual piece. The resulting sound and animation illustrated the exponential rise of climate change, resonating universally when I presented it in both China and Geneva. The striking visual curve of climate change mirrored the exponential growth of AI, emphasizing the urgency of responsible AI development. This project demonstrated how AI can be a force for good, fostering deeper understanding through creative exploration.’
This project demonstrated how AI can be a force for good, fostering deeper understanding through creative exploration.
Express yourself in profound ways
As a conclusion, Portrait XO says to Nxt Museum director Merel van Helsdingen: ‘I’m grateful to Nxt Museum for creating a space that embraces the convergence of innovation, art, and music. As a musician stepping into the new media art space, I’ve found it fascinating to navigate the tension between contemporary art and new media art, as some still don’t view the latter as serious art. Despite this, I ground myself in my identity as a musician, incorporating visuals as an essential part of storytelling both on and off stage. The availability of innovative tools and plugins now allows artists to express themselves in profound ways that resonate with current global challenges.’