Output details

Simulating Synesthesia in Spatially-based Real-time Audio-Visual Performance

Following his review of abstracts Gibson co-edited 1/3 of the papers for this 24-paper volume. Gibson's own paper described the history of audio-visual art and also analysed two of his own works in the area.

This Leonardo publication collects papers on real-time visuals as discussed by film and video theorists, interaction designers, Visual Jockeys (VJs), game programmers, and information architects. Leonardo is a long established international venue for the presentation of research in art and technology. The volume contains a variety of perspectives on live visuals and is the first collected academic volume to cover the area of VJing and real-time visuals. The research imperative was to demonstrate how real-time visuals have a profoundly different quality and therefore distinct requirements from linear visual forms such as narrative film. The volume outlines a radical paradigm in which the moving image is re-conceptualised as a performative medium. The collected body of research in the volume effectively contributes to the establishment of best practice for this evolving form.

The aim of Gibson's paper was to establish how synesthetic simulation can assist user interaction in real-time 3D spatial environments. Gibson's projects deploy motion-tracking technology to enable users to interact with sound, light and video using their movements. The paper demonstrates how motions can be most effectively mapped to consolidate the roles of DJ, VJ, dancer and light designer in one interface. This interaction model simulates the effect of synesthesia. The paper refers to the latest scientific studies on synesthesia in which brain scans confirm cross-modal activity in the brains of synesthetes. The paper provides a history of synesthesia and the arts, and provides examples of how synesthetic mapping is used in Gibson's practice. Extending from this research the paper proposes a new type of synesthesia called ‘Optophonokinesia’ (motion-sound-vision). The paper forwards the notion that the use of simultaneous mapping of tactile, auditory and visual parameters is a powerful tool for establishing a deeper understanding of multidisciplinary art work for audiences and users.