Audio-visual Animation

We all have experienced unlimited illusions when listening to music. A number of artists have devoted to visualize this kind of illusions by producing dynamic images in the form of lines, graphics or color tones. In this regard the music itself is decorated with visual elements. This provides a new space for animation production.

Concept of Audio-visual Animation

Audio-visual animation (one kind of experimental animations) is an emerging animation form with strong visual effects. It contains diverse patterns of visual changes, organizing narrative structure in a nonlinear manner. Rather than act as background in the mainstream animation, music in an audio-visual animation has a dominant position and even plays a role as screen writer. Elements that appear in audio-visual animation are designed depending on designers’ understanding of the music, which is different from the traditional production process of “script-painting-animation”.

Audiovisual Awareness in Animation

Making a motion picture of the changing light and shadow, lines and shapes, colors and shades according to abstract geometry or mathematical formulas to concretize the music is a prevalent way in audiovisual field, where music would present its own character. Sound itself is a dynamic form of art, the structure of which shares very similar characteristics with dynamic images. The soundtrack of  early animations is used to make up for the dialogue insufficiency or enhance the fun of characters’ look and pace, in which case the sound is chosen depending on the development of storyline, only playing a role in expanding the plot or regulating the atmosphere. In comparison, audio-visual animation has more strict requirements for sound. Since the design inspiration could come from one’s perception of sound, audio-visual animation designers prefer to use the sound that is highly capable of creating mental pictures.

With new innovations in technology, audiovisual is evolving from the music video to large, site-specific projections live shows. Although it may seem like a recent phenomenon, its history goes back much further. Oskar Fischinger—an early 20th century German painter and pioneer animator who is regarded as the Father of Visual Music—was specialized in avant-garde, audiovisual films and noted for his abstract shapes synchronized to music. Many of his animations were created using unusual materials such as colored liquid, filters, slides, and wax. He created Studies when he was working on special effects for the movie Woman in the Moon. Studies is a series of abstract short films that feature black and white forms that are synchronized to music. An excerpt from Studie nr. 8 is shown below.

Studie nr 8 (excerpt) by Oskar Fischinger

Someone may argue that the general public  can’t get abstract art, while as long as little shapes swoop or tremble or dance to music, they seem to have purpose and meaning. Therefore Fischinger never tried to illustrate music, but to provide a visual equivalent. He created critically influential animations that combined his strong sense of audiovisual awareness. His method of conveying sound and rhythm with color and shapes is the basis of visual music.

Methods in Making Audio-visual Animation

Running the animation in accordance with the laws of sound is a new method worth exploring. Admittedly visualizing the auditory information by creative design could not only eliminate the pressure of script writing, but also lower the design limitations and may lead to uniquely amazing effects. Compared to the animation with a fixed storyline, the idea of audio-visual conversion and integration is more free to achieve improvisation. Additionally different people have different understanding and imagination of sound, thus to some extent the overlap of script and design ideas could be avoided effectively.

Selecting the sound that is highly capable of creating mental pictures is quite critical in this procedure, which requires us to explore and accumulate a variety of interesting sound, or to record the sound in daily life and then edit them through audio softwares. This is an exploratory and experimental process in training our sensitivity and appreciation for sound, and there are lots of ways to do this. For instance, we could try to make up our own words to some familiar melody, collect some sound that is not appealing to us and then change its use of space to achieve harmony, or add some sound effects to a piece of animation in an unconventional approach.

Character setting of most mainstream animations is based on human body or organism, whereas in audio-visual animation these characters could be replaced by abstract visual modeling. We could generate our own animation style by applying a series of basic elements, making the originally meaningless graphics obtain wonderful results in the rhythm. For example, monotonous and simple abstract graphics could be associated with soft or alienated sound, while sharp graphics would match intense sound. Audio-visual animation basically explores the animation effects in terms of  abstractive and realistic, transformation and deformation, etc..

Overall, audio-visual animation requires designers to intensively research the functions and features of sound as well as the motion characteristics of graphics, which can only be achieved after a long-term artistic experience and design training.

Reference:

Robertson R. Eisenstein on the audiovisual: the montage of music, image and sound in cinema[M]. Tauris Academic Studies, 2009.

Oskar Fischinger: the animation wizard who angered Walt Disney and the Nazis  www.theguardian.com/artanddesign/2013/jan/09/oskar-fischinger-animation-disney-nazis

Women in the moon en.wikipedia.org/wiki/Woman_in_the_Moon

 

 

 

 

 

 

 

 

 

 

Video montage

In the field of audio visual ensemble, it’s always hard to decide whether the sound designers or visual designers should be first to make the next decisive audiovisual step. There would be two options, that is, the sound designers write the music ahead having seen the relevant section of unedited visual content, or to compose it once the visual designers have already edited it. In terms of our AVE group we’ve decided to present our work in the form of a live performance, which requires both sound and visual designers to manipulate their instruments in real time rather than to project the already-integrated work while standing aside. Therefore the first choice seems more feasible for us, which requires visual designers to develop the montage from sound designers’ existing score.

This procedure involved the sound designers (Russell and Timo) drawing the ‘architectural’ plan itself, which would then determine how the audiovisual montage of visual designers (Jessamine and I) would operate. Responsible for the video part in our group I have to create a video montage to match Russell and Timo’s sound cues as well as the whole theme. Here are the main steps of this procedure.

1.  I’ve made a series of video clips which match the theme Cave, using softwares such as      After Effects and Premiere.

video clips_01 Trapped in the cave

video clips_01 Trapped in the cave  vimeo.com/92611381

video clips_02 Go out of the cave

video clips_02 Go out of the cave  vimeo.com/92611782

video clips_03 Outside of the cave

video clips_03 Outside of the cave  vimeo.com/92612143

video clips_04 Outside of the cave

video clips_04 Outside of the cave  vimeo.com/92612228

video clips_05 Outside of the cave

video clips_05 Outside of the cave  vimeo.com/92612279

video clips_06 Outside of the cave

video clips_06 Outside of the cave  vimeo.com/92612428

video clips_07 Go back to the cave

video clips_07 Go back to the cave  vimeo.com/92612493

video clips_08 Struggle in the cave

video clips_08 Struggle in the cave  vimeo.com/92612641

video clips_09 Struggle in the cave

video clips_09 Struggle in the cave  vimeo.com/92612759

video clips_10 The ending

video clips_10 The ending      vimeo.com/92612824

2.   I must have in memory and be very familiar with all the video clips which can be                   manipulated and transformed at any point. With a sophisticated Max patch (thanks to         Martin) and an external controller I could modify many parameters of the videos in real-       time such as frame mangling, brightness, contrast, blur, colour, scale, etc..Moreover all       the parameters could be set to audio-controlled, which is of crucial importance in an           audio-visual-ensemble live performance.

Control the video using Max

Control the video using Max

Max patch interface

Max patch interface

3.  Attention to details during rehearsals and listen again and again to the recorded sound        of the rehearsal video, until the moment arises when I can imagine a series of images          which could correspond with the sound, and realize the effect I had in mind in the next        rehearsal. During this process the matching of the sound cues with corresponding              section of visual representation has produced a satisfactory harmony to some extent.

AudioVisual Study: Silent Rhythm

CONCEPT

We all have the experience that some pieces of music remind us of some particular scenes, we can imagine the picture in head without watching a real image. Then what about the opposite? Could we imagine the sound when watching silent videos? Also If you’re reading this sentence silently, chances are that you’re imagining a voice speaking the words you are reading. Watching videos could be viewed as another form of reading, where words are replaced by a series of images. It’s conceivable that sound could be formed in head even when we are watching silent videos.

AUDIOVISUAL INTERPRETATION

We know that sound is a vibration that propagates as a mechanical wave of pressure and displacement, through some medium. Imaging that we are in outer space, we can feel the vibration while sound cannot be heard because no air as medium. While we cannot hear sound doesn’t necessarily mean we cannot feel it. Some findings suggest that the experience deaf people have when ‘feeling’ music is similar to the experience other people have when hearing music. The perception of the musical vibrations by the deaf is likely every bit as real as the equivalent sounds, since they are ultimately processed in the same part of the brain.

For this study some experimental video clips are made to investigate the correlation of auditory and visual senses. Obviously vibrating objects are of crucial importance in these  videos. I started by making 3D models in Blender, where the vibration effect is achieved   by baking sound to F-curve (one function in Blender). The sound source is <Pulse> by RTPN.

3D cubes

Here is the performance of these cubes. The sound is not exported.

This video is of strong rhythm, which is reflected by the fierce vibration, the changing sizes and colors of cubes. As expected I feel a sound of strong beats expressing excitement, or thrill in my head when watching this video.

I moved forwards to get some inspiration from real stuff in daily life. We already have a sound-assets library that is built through our life experience. We know what the sound should be like when waves lapping the shore, Ping-Pong’s bouncing on the floor, someone knocking the door…When these kinds of scenes appear on the screen we can imagine the sound even if the video itself is silent.

The following video is inspired by tunnel — the real tunnel and the time tunnel. It makes me feel that I’m in a car on the run while I can’t escape anyway. The imaginary sound is in a tremor because of fear. While this video is kind of abstract and audience may not feel any specific sound in head, which makes me realize that concrete objects like waves and explosion maybe better to present sound as well as produce audiovisual effect.

Depending on the former experimental videos, I decide to combine the 3D models and real scenes. As Cinema 4D files can be perfectly imported to After Effects CC, I chose C4D rather than Blender to make models such as vibrating balls, waves, smoke, etc..

屏幕快照 2014-02-27 下午9.46.51屏幕快照 2014-02-27 下午9.42.27屏幕快照 2014-02-27 下午9.40.00

Sound is filtered, whereas the balls in the video are of strong rhythm and scenes switch frequently, both of which create a feeling of intensity. Also you can imagine the sound when the huge ball falls to the ground and the floods submerge the screen. It’s interesting that the video is silent while you cannot feel any peace.

Software used: Cinema 4D, AfterEffects, Photoshop, Premire.

Plugins: Red Giant.

Study:

CONCLUSION

During this journey of “silent rhythm”, you must have experienced the feeling of sound-in-head. Sound is everywhere, sound keeps popping to head unconsciously even you are in quiet conditions, sound has specific beats when you are watching silent videos alive with rhythm. There are a thousand kinds of sound in a thousand audiences’ head.

FURTHER INFORMATION

Although I’m satisfied with the results of the videos, it has to be noted that the process is time consuming, including the 3D modeling, video editing and rendering. Especially considering the interactive requirements for our project’s final performance, the  operability and  conformability of the work is open to question. I need to think about more practical and efficient ways to present the idea and make the work fits a live performance environment. Regardless of the forms, the more important thing is the idea behind these videos — what is silent rhythm, and how silent rhythm can be “heard” and felt by people. I’m keen to seeing how this idea can cooperate with the other group members’ work.

Silent Rhythm

These days I’m thinking about the silent rhythm stuff. We all have the experience that some pieces of music remind us of some particular scenes, we can imagine the picture in mind without watching a real image. Then what about the opposite? Could we imagine the sound when watching silent images or videos? Chaplin’s films give an example. Maybe the idea of “silent rhythm” is kind of off the topic and quite different from your live studies of AVE, I think it’s worth studying to better understand the relationship between audio and visual.

We know that sound is a vibration that propagates as a mechanical wave of pressure and displacement, through some medium. Imaging that we are in outer space, we can feel the vibration while sound cannot be heard because no air as medium. While we cannot hear sound doesn’t necessarily mean we cannot feel it. Some findings suggest that the experience deaf people have when ‘feeling’ music is similar to the experience other people have when hearing music. The perception of the musical vibrations by the deaf is likely every bit as real as the equivalent sounds, since they are ultimately processed in the same part of the brain.

For us normal persons, we already have a sound library that is built through our life experience. We know what the sound should be like when waves lapping the shore, ping-pongs bouncing on the floor, someone knocking the door…When these kinds of scenes appear on the screen we can imagine the sound even if the video itself is silent.

The video provided below is my work for silent rhythm. Sound is filtered, whereas the balls in the video are of strong rhythm and scenes switch frequently, both of which create a feeling of intensity. It’s interesting that the video is silent while you cannot feel any peace. Another option for performance is that live sound could be inserted in this video in a “on and off” form, that is, the sound stops time to time and when stopped the objects keep vibrating, audience may feel that the sound is still there because of auditory inertia (not sure whether this phase exists).

Softwares related: Cinema4D, AfterEffects, Photoshop.

Silent Rhythm

Video Recording at Cramond on 16.02.14

Marco and I went to the beach last Sunday to record some interesting stuff related (maybe not) to AudioVisual. We shot the road surface and the forest when in a driving bus. As there were no big waves as expected we recorded the small ones lapping the shore, and some objects on the sea surface moving along with the wave. We also made use of the sand, leaves, shells, stones and dead trees to make different kinds of sound.

I thought these video materials will not be used for Submission1 so I haven’t edited them. Maybe I will work on them the next few days. I will bring them to our meeting this Friday, feel free to get them then!

The video linked is from one of this recordings, where Marco was giving a live performance with dead tree and leaves^_^The original 1080P files are too large so I have to upload to Vimeo. When I’m writing this post the video is still waiting in line to be converted, hope you can successfully get access to it.

vimeo.com/87137256