Doox's Invitation

« MOD Game : Ask Mr. Hole | Main | Playtest Review - Life By The Fire / Chong Qing »

October 03, 2005

Open Mic. - "Interactive Visual Music" by Doox

“The dream of creating a color music for the eye, comparable with auditory music for the ear, dates to antiquity…”
- William Moritz

I always had two ideas for my final thesis. One is about a narrative game that makes audience feel strong relationship with NPC and the other one is a performance art that combines playing the keyboard with visual arts.

Both Music and Visual Arts are perfect forms of arts by themselves. They have their own depth and beauty. But actually we meet arts which are the mixture of both of them more often than separate one. Mixed form of the arts evokes totally appreciation different to the experience from the separate one. How Music and Visual Arts are affecting each other in creation and combined into one art form and what is the synergy effect by that are one of the most interesting research subjects for me. When I was learning how to play the piano I found that the arrangement of the keys in keyboards helped me remembering different scales. When it comes to learning perfect pitch, there was an exercise that imagining a correspondent color for a specific note. Like that using synaesthesia in education is also one of my matter of concerns.

When I came up with this topic “Interactive Visual Music”, it immediately reminded me of the well-known popular film “Fantasia.” So my first idea out of it was sort of “Real-time Interactive Fantasia.” Also visualization tools in Windows Media Player and WinAmp might be the closest art form to my envisioning of “Visual Music.” Fantiasia and Visualization tool are very different to each other in their creation and appreciation. Fantasia is an animation art created on the basis of well-known classical music pieces. In its order of creation, music is first, and then the animation is carefully created inspired by the pre-made music. In terms of the inspiration from one art from the other, Fantasia is one-way. The visualization tools I mentioned above are also one-way in its creation. In visualization tools, although the visual scenes are created procedurally which is different to Fantasia, only music affects the creation or modification of the visual scenes and visual scenes doesn’t have anything to do with creating music at all. My envisioning of “Interactive Visual Music” is two way form that music and visual scenes affect each other when they are created in real-time. The degree of affecting each other can be different but both of them must have some relationship at the point of its creation. That means when creating a mixed form of the arts, the artist should be inspired by both of the art forms in its progress of creating or performing the mixed form of the art.

When I was thinking about specific interface for this art I instantly came up with this idea of having musical keyboard connected to PC by MIDI interface. Keyboard is the best musical instrument that can express broad range of complicated musical elements for single player. That is why it is considered necessary for composers to learn how to play the piano. I need to program a visual music performance application which makes it possible to visualize what a keyboardist plays in real time by analyzing actual MIDI signals into various musical elements like pitch, harmonics, velocity of pushing keys, types of musical patches and so on. With those analyzed data out of MIDI signal, the application will control various visual elements on the screen like types of object, color, scale, animation, etc. The application will offer various options for keyboardists to preset what kind of musical elements will take effect on what kind of visual element. Main difference compared to other music visualization tools I mentioned before is that my visualization program will use actual musical elements, which I mentioned above, out of MIDI signal; which is different from using sound frequency of the music being played by the application. Especially with great improvement of today’s 3D engine it’s possible to render real-time high polygon objects with various effects almost close to that of cinema in its detail.

Preparing this open mic. I could find that “Visual Music” has already a long history.
A good start point to find various forms of “Visual Music” was “Center for Visual Music”.
http://www.centerforvisualmusic.org/

Also “iota center” is one of the major sites for “Visual Music”.
http://www.iotacenter.org/

The famous visual music artist Oskar Fishinger’s site.
http://www.oskarfischinger.org/

William Moritz’s article about “The Dream of Color Music, And Machines That Made it Possible”
http://www.awn.com/mag/issue2.1/articles/moritz2.1.html

When I found a visual artist from various internet sites who’ve had same idea with me and had already developed the idea, I felt like meeting an old friend of mine.
Especially for the idea of using musical keyboard as an input, I found there is someone who had already done similar work before. Like the perfect pitch exercise I mentioned before, the way it visualizes the music is mostly like expressing corresponding color to a specific note or harmony.
http://homepage.tinet.ie/%7Emusima/visualmusic/visualmusic.htm


It is kind of limited way of visualization compared to my picture, though. But now, thanks to today’s great improvement of PC, real-time graphics opened a whole new era for individual artists and I want to make the best use of that in my visual music work.

This link below is about Audiovisual work from MIT guys. It was quite different approach because it has visual art tool as an interface in creating certain audio work. Don’t miss the movie files from the link.
http://acg.media.mit.edu/people/golan/aves/


This work is perfectly two way creation, procedural contents and real-time performance. The content for this work is totally abstract. For me, audio for this work is more like sound effect than music. This work is the closest one to my picture of visual music except the fact that it uses visual art tool as an interface and it doesn’t actually have something to do with musical elements I am envisioning.

Watching various kinds of visual music arts, I had some questions for myself about them.
1. In the process of creating music or visual arts, is either of them actually affected by the other? ( Most of the visual music works I could meet were one-way which means one art form is made on top of a pre-made art form inspired by it. Like animation made out of a music and so on. )
2. Does it contain procedural contents or not?
3. Is it real-time or prescripted?
4. What’s the interface for the creator or author? Is it making a visual out of an musical interface(or instrument) or creating a music out of an art tool?
5. Is the content containing obvious narrative? Or is it more abstract and symbolic?
6. What are the actual musical elements or visual elements that affect each other in their creation?
7. Does it still have enough meaning or appreciation when it’s experienced separately?
8. What are the synergy effects when they are combined together?

I think this is a great beginning for me to start researching this area, and I am happy to find that I am living in a capital city of “Visual Music”. Lots of visual music artists and important organizations are here in LA. I wish to experience more visual music works from now on and keep developing my own idea. Visual music is also really fit to my study on programming procedural 3D graphics. I hope to start developing something on this as soon as possible. That leaves me lots of study in music, arts and programming. How exciting!!!

Posted by doox at October 3, 2005 11:30 PM

Comments

Post a comment