Seeing Echoes in the Mind of the Whale

Seeing Echoes in the Mind of the Whale explores the sensory world of cetaceans and celebrates the multitude of life forms that make up our Earth’s oceans. Blending extensive hydrophone recordings, underwater videography, scientific insight, and real-time computing, Seeing Echoes in the Mind of the Whale offers a multisensory experience that seeks to bridge the gap between our world and the whales’ umwelt.

The piece follows the journeys of three main species—a bottlenose dolphin, a humpback whale, and a sperm whale. Each of these creatures take a breath at the water’s surface before diving back into the deep blue. With each breath, the perspective shifts, allowing viewers to experience the world through the eyes and senses of these extraordinary animals. Through this shifting lens, the installation highlights the unique ways these species perceive their surroundings.

The project was created in collaboration with Tom Mustill and with the scientific support of the UPC Bioacoustic Applications Laboratory.

Commissioned by Fundación Telefónica and DHub (Disseny Hub Barcelona) 

Marshmallow Laser Feast conducted thorough research guided by the wonderful storyteller, filmmaker, and author, Tom Mustill and marine biologist Steffen De Vreese of UPC Bioacoustic Applications Laboratory. We’ve collaborated with marine biologists and bioacousticians to explore and interpret the sensory world of cetaceans. Through a series of interviews and an exhaustive study of available scientific literature, the team gained insights into how these marine species perceive and interact with their environment in ways that go beyond human senses.

The soundworld of the project is built on raw whale vocalizations primarily provided by the Monterey Bay Aquarium Research Institute and the Laboratory of Applied Bioacoustics at the Polytechnic University of Catalonia. These recordings, captured from oceans worldwide, undergo advanced signal processing and machine learning techniques to identify and categorise individual species’ vocalisations by their respective institutes. These algorithms allow the researchers to sift through vast datasets, isolating specific sounds and behaviours that form the basis of the auditory narrative.

The visual element is constructed using a custom generative AI pipeline designed to extract features from underwater footage such as corals, dolphins, humpback whales, sperm whales, and other marine elements, segmenting them and reconstructing volumetric data for each scene. This enabled us to merge disparate video sources into a coherent, immersive underwater world.

This data is then processed through a real-time render engine that uses auditory data as a dynamic input to manipulate the visual environment, creating a speculative rendition of sculpting with sound. The resulting multi-channel video and audio outputs are rendered into an immersive installation, allowing us to glimpse what it might feel like to “see” through sound, as whales do.

The installation is structured around three key species, each serving as a guide through their own chapter.

1-Bottlenose Dolphins: Sculpting with Sound

The first chapter focuses on Bottlenose Dolphins, whose remarkable sensory experience is dominated by echolocation. Unlike humans, these dolphins process acoustic information in their visual cortex, creating a unique sensory fusion where sound becomes a sculpting tool for imagery. Their world is a juxtaposition of sound and sight, where echolocation allows them to “see” through echoes, forming a dynamic, three-dimensional understanding of their surroundings.

Dolphins have colour vision, though it differs from most land mammals. They can perceive shades of grey and are sensitive to near ultraviolet and blue-green light. Genetic studies show that their rods and cones are adapted for underwater environments, allowing them to see with a blue-shifted spectrum compared to terrestrial animals. These adaptations highlight their evolutionary transition from land to sea.

2-Humpback Whales: Depicting memories of the ocean through songs

In the second chapter, Humpback Whales take centre stage, renowned for their hauntingly beautiful songs that echo through the ocean. These complex vocalisations dominate the deep blue, creating an acoustic tapestry that spans vast distances.  Their songs, steeped in mystery, form the heartbeat of the ocean’s soundscape.

3- Sperm Whales: Scanning through space and multi-temporality with sound. 

The final chapter brings focus to the incredible Sperm Whale, whose vocal abilities surpass any other. Sperm whales can navigate the shadowy depths of the ocean, where sunlight cannot reach, using their powerful vocal clicks to communicate across immense distances. Their echolocation abilities not only guide them through the darkness but also enable communication that can cross entire oceans, connecting them to each other in ways still not fully understood by humans.

Each chapter highlights the profound ways these species experience their environment, using sound as their primary sense to shape their world in ways we are only beginning to grasp.

Seeing Echoes In the Mind of the Whale, 2024

An Artwork by Marshmallow Laser Feast (Ersin Han Ersin, Barnaby Steel, Robin McNicholas)

In collaboration with Tom Mustill and with the scientific support of the UPC Bioacoustic Applications Laboratory.

Commissioned by Fundación Telefónica and DHub (Disseny Hub Barcelona)

Executive Producers: Carolina Vallejo, Eleanor (Nell) Whitley, Mike Jones.

Artist and Lead Developer: Natan Sinigaglia, Senior Producer: Martin Jowers

Sound Engineer and Spatialisation: Simon Hendry, VFX Artist: Nicolas Le Dren

Infrastructure Lead: Andrew Robinson, Head of Studio: Sarah Gamper Marconi,

Gen Ai VFX Pipeline: Nilor Corp

Scientific Advisors & Contributors, Video and Audio Sources 

Professor Michel André (LAB), Steffen De Vreese, MVM, PhD (LAB), Tom Mustill, John Ryan (MBARI), Howard Hall, Steve McNicholas & Luke Cresswell (Yes/No Productions), Daan Verhoeven, Misael Morales Vargas (Biosean), Open Planet (openplanet.org)

Dedication

In memory of Karen Bakker, whose insight, intellect and writings have been a profound source of inspiration to us.