Audio-visual Animation

We all have experienced unlimited illusions when listening to music. A number of artists have devoted to visualize this kind of illusions by producing dynamic images in the form of lines, graphics or color tones. In this regard the music itself is decorated with visual elements. This provides a new space for animation production.

Concept of Audio-visual Animation

Audio-visual animation (one kind of experimental animations) is an emerging animation form with strong visual effects. It contains diverse patterns of visual changes, organizing narrative structure in a nonlinear manner. Rather than act as background in the mainstream animation, music in an audio-visual animation has a dominant position and even plays a role as screen writer. Elements that appear in audio-visual animation are designed depending on designers’ understanding of the music, which is different from the traditional production process of “script-painting-animation”.

Audiovisual Awareness in Animation

Making a motion picture of the changing light and shadow, lines and shapes, colors and shades according to abstract geometry or mathematical formulas to concretize the music is a prevalent way in audiovisual field, where music would present its own character. Sound itself is a dynamic form of art, the structure of which shares very similar characteristics with dynamic images. The soundtrack of  early animations is used to make up for the dialogue insufficiency or enhance the fun of characters’ look and pace, in which case the sound is chosen depending on the development of storyline, only playing a role in expanding the plot or regulating the atmosphere. In comparison, audio-visual animation has more strict requirements for sound. Since the design inspiration could come from one’s perception of sound, audio-visual animation designers prefer to use the sound that is highly capable of creating mental pictures.

With new innovations in technology, audiovisual is evolving from the music video to large, site-specific projections live shows. Although it may seem like a recent phenomenon, its history goes back much further. Oskar Fischinger—an early 20th century German painter and pioneer animator who is regarded as the Father of Visual Music—was specialized in avant-garde, audiovisual films and noted for his abstract shapes synchronized to music. Many of his animations were created using unusual materials such as colored liquid, filters, slides, and wax. He created Studies when he was working on special effects for the movie Woman in the Moon. Studies is a series of abstract short films that feature black and white forms that are synchronized to music. An excerpt from Studie nr. 8 is shown below.

Studie nr 8 (excerpt) by Oskar Fischinger

Someone may argue that the general public  can’t get abstract art, while as long as little shapes swoop or tremble or dance to music, they seem to have purpose and meaning. Therefore Fischinger never tried to illustrate music, but to provide a visual equivalent. He created critically influential animations that combined his strong sense of audiovisual awareness. His method of conveying sound and rhythm with color and shapes is the basis of visual music.

Methods in Making Audio-visual Animation

Running the animation in accordance with the laws of sound is a new method worth exploring. Admittedly visualizing the auditory information by creative design could not only eliminate the pressure of script writing, but also lower the design limitations and may lead to uniquely amazing effects. Compared to the animation with a fixed storyline, the idea of audio-visual conversion and integration is more free to achieve improvisation. Additionally different people have different understanding and imagination of sound, thus to some extent the overlap of script and design ideas could be avoided effectively.

Selecting the sound that is highly capable of creating mental pictures is quite critical in this procedure, which requires us to explore and accumulate a variety of interesting sound, or to record the sound in daily life and then edit them through audio softwares. This is an exploratory and experimental process in training our sensitivity and appreciation for sound, and there are lots of ways to do this. For instance, we could try to make up our own words to some familiar melody, collect some sound that is not appealing to us and then change its use of space to achieve harmony, or add some sound effects to a piece of animation in an unconventional approach.

Character setting of most mainstream animations is based on human body or organism, whereas in audio-visual animation these characters could be replaced by abstract visual modeling. We could generate our own animation style by applying a series of basic elements, making the originally meaningless graphics obtain wonderful results in the rhythm. For example, monotonous and simple abstract graphics could be associated with soft or alienated sound, while sharp graphics would match intense sound. Audio-visual animation basically explores the animation effects in terms of  abstractive and realistic, transformation and deformation, etc..

Overall, audio-visual animation requires designers to intensively research the functions and features of sound as well as the motion characteristics of graphics, which can only be achieved after a long-term artistic experience and design training.


Robertson R. Eisenstein on the audiovisual: the montage of music, image and sound in cinema[M]. Tauris Academic Studies, 2009.

Oskar Fischinger: the animation wizard who angered Walt Disney and the Nazis

Women in the moon











2 thoughts on “Audio-visual Animation

  1. Hi Shuman, this is very interesting and we both wrote about mental images. Can I ask you what do you exactly mean by “collect some sound that is not appealing to us and then change its use of space to achieve harmony”?

    • Basically I mean that, in daily life when we hear some sound that we don’t like (for example, hearing sharp sound when reading a book), we just ignore that sound or leave that environment. While it does not necessarily the sound itself that takes the blame, if we could put the sound in another environment we may find it works well and even shows surprising effects beyond expectation. It requires imagination and association^_^

Leave a Reply