GridJam

A 21^st Century Virtual Color Organ Project

By Jack Ox and Dave Britton

Summary:

GridJam will be a multi-site international performance of electronic music, compositionally scripted with improvisation, incorporating real-time music visualization and video avatars, performed and presented in immersive virtual reality and broadcast video over the Internet2 Access Grid.

General description:

GridJam will be a production of the Virtual Color Organ^TM visualizing Alvin Curran's electronic music performed by musician located at multiple places. The Virtual Color Organ^TM (VCO) is a 3D immersive environment (i.e. a "virtual reality" installation) in which music is visually realized in colored and image-textured shapes as it is heard. The visualization remains as a 3D graphical sculpture after the performance. The colors, images, shapes and even the motions and placement of the visualized musical shapes are governed by artist-defined metaphoric relationships, created by hand as aesthetic and symbolic qualities rather than algorithmically. The VCO visually illustrates the information contained in the music's score, the composer's instructions to the musicians, and the musicians contributions to the score as they improvise in reaction to each other's performances and to the immersive visual experience. Illustrative of synesthesia and intermedia, the VCO displays the emergent properties within the meaning of music as information.

The aesthetic vs. algorithmic quality is exemplified in the extremely high-resolution detailed hand-drawn imagery of desert landscape formations that comprise the current "visual organ stop" to be used for GridJam. The images are designed to provide suggestive metaphors for the representation of timbre-based contemporary music. In this case, composer Alvin Curran's library of nearly two hundred found sounds provides the starting point for the electronic composition. For GridJam, each of these sound files is provided with a hand-made 3D object whose shape is based on the amplitude and pitch waveforms of the sound file's audio analysis.

GridJam is a geographically distributed performance. The musicians are placed in their own immersive displays in different locations, and each plays a MIDI-equipped (Musical Instrument Digital Interface) instrument. Each musician is videoed from three angles, to construct a 3D video avatar.

The resultant data- the performers' MIDI data streams and the video signal - is processed and distributed. The video is first combined and processed into a dynamic 3D avatar object using software developed by Pierre Boulanger at the University of Alberta in Canada. The avatar video and the MIDI data are sent from each performer's station to all of the others. At each location, the incoming data is combined with the locally generated data, and then fed to the Color Organ software. The combined immersive audio-visualization is rendered at that location. At arbitrary and moveable virtual camera positions in the immersive display, images of the combined performance are captured and mixed with the audio to create a non-immersive audiovisual stream for broadcast to audiences outside the immersive environment.

Audiences will see the video presentation conventionally on closed circuit TV, and may also be able to see it streamed on the Internet.

Audiences within the immersive environment will be able to arbitrarily move around within the 3D world watching the avatars and the music-shapes as they listen to the music.

Developmental partners:

• EAI/Sense8 ' World Toolkit graphics library software
• Silicon Graphics' Infinite Reality Engine computer
• Ars Electronica' initial prototype production seed funding
• University of Illinois National Center for Supercomputing Applications (NCSA)' software and technical assistance
• University of New Mexico, Art Technology Center and Albuquerque High Performance Computing Center' Artist in Residence grants, production support
• University of California at San Diego, Center for Research in Computers and the Arts (CIRCA) and California Institute for Telecommunications and Information Technology Cal(IT)² 's production support
• University of Alberta - Pierre Boulanger, dynamic 3D video avatar software
• Boston University, Center for High Performance Computing and Communication for the Arts' audio and network technical assistance

Technical structure:

The GridJam consists of an arbitrary number of networked nodes, each of which is a system of one or more computers providing the following services:

• Real-time immersive display of the VCO world,
• Receiving and processing video avatar streams from other nodes and displaying them in the VCO world,
• Digitizing and processing video camera data in real time into the local node's video avatar, displaying this avatar image locally within the VCO world and transmitting it over the network,
• Receiving and processing of MIDI data in and out of the network,
• Generating very high quality audio based on MIDI input data, including applying real-time transformations to the basic "found sound" library. The real-time MIDI performance data will indicate which sounds to play, which transforms to apply and the degree or amount of each transform to apply.
• Processing local viewpoint control data, to change the viewpoint arbitrarily in response to a mouse, wand, joystick or other controller,
• Building video streams from virtual camera viewpoints within the VCO world and transmitting them for network broadcast, including receiving and processing instructions to move or reposition the virtual camera viewpoints.

These nodes are attached to the Access Grid for lowest possible latency in transmissions among each other. The data being transmitted consist of the low bandwidth MIDI control signals and the high bandwidth video avatar streams, as well as the high bandwidth captured video imagery from the observational viewpoints.

We expect to have at least six distinct immersive nodes, with a minimum of four performers in total.

The artists, critical and conceptual foundations, and project history:

Jack Ox and Dave Britton began working together in 1998, following Ox's receipt of a commission from Ars Electronica to prepare a prototype example of how she could adapt her twenty years of experience producing 2-D work in the visualization of music to the immersive 3D environment. Britton came to the project with the experience of having produced the software for the seminal LASCAUX virtual reality installation of the French Paleolithic painted cave, as well as major commercial credits in web site production technology for the original NBC.com, and his own company adapting 3D virtual reality techniques to animation and computer gaming.

Ox felt the extant state of immersive art was abysmal, too cartoony and gimmicky, and filled with images of insufficient quality. With Britton's encouragement and technical support she learned the re-orientation necessary to work in three dimensions, and focused on producing detailed finely drawn images by hand which could be incorporated into the computer imagery. Ox historically prepares a highly complex metaphorically driven system for representing the information from a musical score in strips of color, applied over a symbolically significant background, all painted in exquisite detail. The visualization system is prepared a priori and the musical score is rigorously processed through it. The beauty and inherent patterns of the information within the music create a resonating and synesthetically beautiful work of intermedia art. Ox determined to maintain the same high standards of quality and conceptual rigor as she moved into three dimensions.

Ars Electronica later chose to stop continuing its support past the prototype phase, judging the work "insufficiently interactive", a failure of critical understanding characteristic of the shallow perspective of that time as the dot-com boom was reaching its peak. Other support, however, emerged from the scientific community, whose interest was piqued by the potential to apply an artist's eye and sensibility to the problem of the visualization of data and the interpretation of complex interactive systems. The Institute of Electronics and Electrical Engineering (IEEE) featured the project in its summer 2000 Multimedia magazine edition.

Ox and Britton came to realize that the work did not need to be limited to a single musical composition, but really constituted a "virtual color organ" capable of adaptation and visualization of any number of musical compositions. Working with Clarence Barlow, they produced "Im Januar am Nil", a 1978 composition for small chamber orchestra of Barlow's based on an unusual spiraling temporal pattern. Each time through the central thematic structure the next variation is an algorithmically determined number of quarter notes longer. Although this pattern is difficult or impossible to apprehend by ear, it shows clearly in the 3D visualization.Ê Next, Alvin Curran was selected to provide a compositional structure suitable for a multiple location geographically distributed piece, since Curran is very experienced with this mode. The GridJam is the visualization of his composition, designed for distributed performance.