Audio-Visual Map

Spatial music for VR and AR

Audio-Visual Map is a spatial music composition for both AR and VR. It was created in collaboration with sound designer Lajos Kolozsvári during my short 1 month artist residency programme in Chronus Art Centre, Shanghai (supported by Kitchen Budapest and Summer Sessions).

In big cities the environment is surrounded by prolific information-overflow which influences our travel experience. What is also evident is the shift of attention towards the private digital space of the inhabitant’s smartphones. Audio-visual map is an experiment based on these facts, exploring an alternative audio-visual version of public spaces. Walking in the streets generates a musical composition, an opposite of the linear playing of music associated with 3D visuals. This expands the original function of the space, transforming it into a digitalized audio-visual experience.

week2_3_floorplan with panoramas.jpeg

The AR geolocation based version
can be explored on the actual streets with a smartphone app and earphones. Chronus Art Centre is based in Shanghai's M50 art district, previously it was a textile factory. The structure and size of the district was perfect to start iterating my project. I took panorama pictutres about the area, drew a map and started to highlight point of interests based on the street's characteristics.

Later on I also recorded sound samples on spot and handled them to Lajos. He was working remotely from London based on my photos, descriptions and sound recordings. He composed different atmospheres from the material, and we tried to reflect on the space, extending it's characteristics in to an ambient music composition. We reflected on objects and enviroments such as the sonic sounds of airconditioning,  signs of traffic, jungle atmosphere on the corridor with tons of plants, etc...


As soon as we created few sound sketches I localized them by gps coordinates so I could quickly test them in real space. For the sound engine we used Audiokinetic wwise, because it allows interactivity such as bpm sync, sound bank optimization and better flexibility than Unity's built in sound capabilities. As the amount of sound's grew we had to divide the map to 4 groups (approx. 20 x 4 audio samples) so even older smartphones can handle the amount of sound layers. There were 3 specific types of sounds used on the map: atmospheric loops, trigger sounds, sounds moving around. 

The VR version
was developed parallel with the AR version so I could test the sounds in pure digital space. The structure and units of the buildings were transformed into simplified 3D graphics and an aditional layer of animations was added. An other AR and VR version of the project was created in Budapest's Liberty Square. This time the VR version was created with photogrammetric 3D scanning techniques to explore new visual characteristics of the enviroment.

UI mockup for the AR app.

IxD, Unity Dev: Gabor Pribek, Sound Design: Lajos Kolozsvári, Project Support: Kitchen Budapest, Media Art Gallery: Chronus Art Centre

Shanghai, 2016