Immersive Audio-Vision

Brief

Introduction

Immersive Audio-Vision aims to explore the concept of multidimensional composition through the creation of 3D audio-visual worlds. You will be using Virtual Reality technology to create an interactive experience by bringing sound and image together in a musical way. You will achieve this through the use of a highly versatile and customizable 3D environment such as a game engine.

During the recent years, there have been many developments in field of sound and graphics interaction. As GPUs get more powerful, offline rendering becomes a thing of the past. This project aims to harness this power and combine 3D sound and graphics in an immersive and nonlinear way. For more details on each submission, refer to the respective pages (sub1, sub2). Examples of similar work can be found on the media page. Altogether, the project is very interdisciplinary—students from all backgrounds are welcome.

Some of the themes and areas that will be explored are:

  • Doublethink
  • Immersive media arts
  • Expanded cinema
  • Nonlinearity
  • Timeless composition
  • Disembodied performance

 

Aims:

  • To develop skills in using game engines as compositional tools.
  • To gain insight into creating 3D audio-visual compositions.
  • To explore immersion through designing meaningful and dynamic relationships between sound and image.
  • To develop a deeper understanding of nonlinearity in composition.
  • To experiment with the role of interactivity in multimedia arts.
  • To push the boundaries of VR development and experiment with the notion of abstract virtual realities.
  • To organise and execute a demanding project though effective collaboration and management of time.

 

Form

It will be up to you as participants to decide how you want to present the project. Perhaps as composition is a center element, it may seem logical to create an installation. While this is true, the project is flexible in that regards and can take any form as long as it brings together the different artistic and technological elements in an expressive way.

 

Possible Roles:

As mentioned above, the project is open to students from all backgrounds. Considering the amount of work, the group will require a minimum of four or ideally six members. The list below contains some of the skills or roles that will be perfectly in line with the project:

  • Game designer
  • Sound Designer
  • Game Programmer
  • Visual Designer
  • Web Designer
  • Audio Programmer
  • Composer
  • 3D Artist
  • Animation Artist
  • Software Engineer
  • Audio Engineer

 

Technical Components:

The project will make use of a wide range of technologies. As the amount time you have is very limited, it’s crucial that you organise a well-defined framework or set of tools that help you complete the necessary tasks. Below are some of the recommended tools and technologies:

  • Unity Game Engine
  • Fmod Audio Engine
  • A 3D Modeling Software (Maya/Blender)
  • A Digital Audio Workstation of your choice
  • Pure Data/Max
  • Oculus Rift VR headset
  • Gear VR (for mobile applications)
  • 3D Audio Tools
Some Suggestions:
  • Share everything with each other from the start—create group chats on a social media platform that suits you best.
  • Break the problems down into smaller chunks and focus on making each component work before worrying about the bigger picture.
  • Actively make blog posts. It is very important that you do this from the start. This will allow you to make better progress and make more sense of the technologies and concepts you are dealing with. The success of your project relies on a strong conceptual foundation.
  • Make manageable deadlines and work to meet them.
  • Divide the tasks based on your skills and  interest.
  • Create prototypes and test them before investing resources into something.
  • Imagine your piece and try to find something similar to examine so that you can reflect critically on your work and others.
  • Read relevant literature and try to understand of your work’s discipline and the audience it may or may not have.
Further Readings:

For related works and inspirations, refer to the media page.

Virtual Reality Musical Instruments: State of the Art, Design Principles, and Future Directions

Rees, A. L. Expanded Cinema: Art, Performance, Film. London: Tate Gallery Pub, 2011.

Gillen, Marilyn. “VR Becomes Art in Hands of Creator.” Billboard 107, no. 41 (1995): 62.

Lokki, Tapio, Jarmo Hiipakka, Rami Hnninen, Tommi Ilmonen, Lauri Savioja, and Tapio Takala. “Realtime Audiovisual Rendering and Contemporary Audiovisual Art.” Organised Sound 3, no. 3 (1998).

Polfreman, R., M. Loomes, and R. Wright. “Physically Inspired Interactive Music Machines – Making Contemporary Composition Accessible?,” 2003.

Peter. Pearce. Structure in Nature Is a Strategy for Design. Cambridge, Mass.: MIT Press, 1978.

Stern, Nathaniel. Interactive Art and Embodiment: The Implicit Body as Performance. Arts Future Book. Canterbury: Gylphi Limited, 2013.

Marselas, Herb. “Integrating Your Game Engine into the Art Pipeline, Part 2: Integrating with Maya.” Game Developer 9, no. 4 (2002): 40–46.

Hyde, Joseph. “Musique Concrète Thinking in Visual Music Practice: Audiovisual Silence and Noise, Reduced Listening and Visual Suspension.” Organised Sound 17, no. 2 (2012): 170–178. doi:10.1017/S1355771812000106.

Mark Stephen. Meadows. Pause & Effect: The Art of Interactive Narrative. Indianapolis, Ind.: New Riders, 2003.

Nathan Cabot. Hale. Abstraction in Art and Nature. New York: Dover, 1993.

Sims, Karl. “Particle Animation and Rendering Using Data Parallel Computation.” ACM SIGGRAPH Computer Graphics 24, no. 4 (1990): 405–413. doi:10.1145/97880.97923.