Output details
34 - Art and Design: History, Practice and Theory
University of Northumbria at Newcastle
Virtual VJ
Virtual VJ is an immersive environment that uses motion tracking technology to allow two users to control sound and video with their movements in 3D space. The focus of the media integration is on the development of observable connections between the audio and video mediums in order to assist the users with ease of interaction. This cross-modal control is made possible by using the camera-based tracking of the Gesture and Media System (GAMS), which can be programmed to allow users to simultaneously control different media elements with their body motions. The interaction model simulates the effect of synesthesia, in which certain people experience colour in response to musical tones. The project has been exhibited in key international venues such as DIS, the world’s premiere interaction design conference.
The key research imperative that is explored in Virtual VJ is how to best enable both cooperation and a sense of personal space in virtual systems. This is achieved by programming the environment so that dramatic events happen when the two trackers are close together or distant. The research process for Virtual VJ began with a study of how several dancers interacted with our tracking system. Their movements were observed and used to make judgments about how to most effectively map space for audio-visual interaction. A detailed list of connections between user movement, sound and video was then developed based on how the mediums could be synesthetically combined most effectively. The project was extensively beta-tested and has been used by expert music performers, dancers and the general public. The project establishes an innovative and rigorous mapping system for the creation of multidisciplinary audio-visual works triggered by tactile motions of users, and has been moved from Canada to the UK to China and Singapore for use with diverse audiences and users.
