media design - mixed reality


Summer Sessions Residency in Chronus Art Centre, Shanghai M50

WEEK 1 - introduction / July 20th - July 24th

I arrived to Shanghai and occupied the accommodation on Wednesday night. Next day I visited Chronus Art Centre and got an introduction to the team and studiospace and also visited some M50 galleries such as ShanghART, CAC and a design shop with expensive but nice pottery and porcelain.

The weekend I was recovering from jetlag, buying some basic householding stuff and visited some city attractions. The life here for a foreigner is difficult, the prices are very high (7th most expensive city in the world), the potatoes are ridiculously expensive (1 euro for 1 piece at the supermarket) and other vegetables as well. It is way harder to get around here because the internet is blocked. The environment is inspiring, tons of supermarkets (few hundred of them) lot of dirt and poor people on the streets selling stuff or even having workshops with sewingmachines, bicycle repairments, cooking in pop-up tents etc. Few old town remains, the biggest one is Yu Garden which is also a big market and used to be a Buddhist temple complex. The cheap food is full of additives to improve the very bad quality ingredients. They state that the city is a hell for poor people and heaven for rich one as luxury services are also widely available. Most of the people have scooters with silent electric engines and it is very dangerous because they drive and slalom on the pathways sometimes with around 30kmph and you cannot hear them. 


WEEK 2 – brainstorm and data collection /  July 25 - July 31

On the second week I almost got recovered from the jetlag and started to finally work on the Audio-Visual Map project. I had a brainstorming with Fito who is a long-term resident artist from Colombia and Cheng who is the director of CAC. Fito is creating crazy machine based interactive installations, he is an expert in Open Frameworks, check-out his website: and Cheng is also in to interactive installations:

The streets of M50 is very suitable for my experiment so I decided to create the prototype there. The Open Street Maps and Google Maps doesn’t include accurate information about the streets here and the Chinese Baidu map would be suitable just for redrawing because I cannot access the SDK of it. The management of M50 has accurate floorplans of the area but they couldn’t give it to me for security reasons. Fortunatley there were many printed, less accurate floorplans about the district which I could redraw in illustrator and later match it with real world and GPS coordinates.

The vector blueprint was a great base for the 3D modeling, so I could create a quick 3D representation of the M50 space with low polygons and simple geometry suitable for smartphone real-time rendering.

I also did 360 photos about the space as google street view is not available here to analyze the space. It was also handful for my sound designer friend Lali, who btw helps to create the soundscapes for this experiment. With the help of these 360 panoramas and the floorplan I cloud explain to him the locations to have a better understanding about the atmosphere and brainstorm online.

My next step was to record good quality stereo sounds with a zoom H2 microphone on spot for the sound design an create the soundscapes inspired by the streets, corners, objects and atmoshperes of M50 which was a textile factory before and they transformed it in to art galleries now.

The end of the week I was struggling with slow internet connections but successfully downloaded the android SDK and set-up the Samsung S4 for development. Had to do a GPS fix with an app called GPS fix and also calibrate the display for my google cardboard.


WEEK 3 - early prototype / Aug 1 - Aug 7

I found a useful asset for Unity3D called MapNav which is kind of a GPS engine for mobile development. It helped me a lot and in a day I could create the first GPS based prototype. It helped to put my custom 3D models to the desired GPS coordinates.  With Unity’s 3D sound engine I could also test some early sounds walking around M50.

After that I tested GoogleCardboard for the VR version but even with a simplified 3D graphics the Samsungs S4 couldn't play faster than 24fps which is not enough towards the desired 60 fps. So I decided to test the OCULUS DK2 which was available at the CAC studio. I had a lot of struggles with the OCULUS SDK, it basically took days do download it and find the right SDK versions for the UNITY 3D as there is no real support for the specific Unity compatibility and the Great Firewall of China was limiting access to the needed websites.. After few days and few OCULUS runtime + Unity integration version tryouts I could finally successfully use it for my project and start optimizing it. By the end of the week I had an Oculus prototype but still was lacking 3D animations and sound integration.


WEEK 4 – presentation week / Aug 8 - Aug 14

The last week was almost without sleep till the presentation. I had to debug the WWISE integration with my friend LALI who did the sound design. It took us few days to find out how to use WWISE for multiplatform (Android, Windows8, Mac). After 24 hours of continuous debugging we finally figured out that the Wwise soundbanks had memory issues on the smartphone and also needed some windows unity assets for the oculus version as I we were developing the whole thing on a Mac. The solution was to use multiple sound-banks and have better memory assistance on the smartphone. After the sound was finally working I had little time to supplement it with 3D animations reacting to the sound. By the end of the week the GPS version of the app was also broken but after recalibrating the unity project file I could use the new Wwise version with both GPS and VR. The presentation was a good experience, Bruce was translating my talking in to Chinese, CAC kept exhibited my project till 6th of September in the HALL of their Gallery and I got an offer from Brew&Blend festival to exhibit the piece. ) .


WEEK 5 – traveling home / Aug 15 - Aug 18
Arrived home and started to focus, polish and further develop the experiment for the AC/DC exhibition at Kitchen Budapest which will happen in late October.

Gábor Pribék