SONANOS: A Musically Structured World
Abstract
SONANOS is an application for interactive visual music. In this PC software, the user simultaneously composes music and constructs objects in a 3D environment. Combining objects generates rhythms, melodies, and phrases, which are determined by the visual properties of the procedurally generated objects. In the virtual environment, users can manipulate both the visual and musical structures. As a result, the user character evolves. The user character transforms, which represents the development of the composition. SONANOS adheres to the aesthetic conventions of the visual music genre, in which abstract graphics correspond to the rhythm. In this project, interactivity amplifies the synesthetic experience by enabling self-expression and deepening the connection between graphics and music.
II. KEYWORDS
SONANOS, synesthesia, interactive visual music, music video game, art game, procedural/generative/algorithmic graphics/ music/animation, music creation/composition game, musically structured world.
III. INTRODUCTION
Visual Music and Abstract Art
When an audience watches visualization of music they naturally search for the connection between the graphics and music. Abstract art offers the audience more opportunities to interpret a meaningful relationship with the music. This interpretation stimulates the minds of the audience. SONANOS introduces interactivity to this art genre in order to amplify the synesthesia. The user not only interprets the graphics, but also constructs them. Through participation, the user expresses his or her mood in multimedia, both visually and aurally.
Interactive Visual Music
Beyond vision and hearing, interactivity engage another sense, which is the sense of touch from the physical interface. Engaging the interactive world with multiple senses naturally immerses and motivates the user. Furthermore, interactivity establishes the opportunity to challenge the user. Also the effort required to mutate the avatar and manipulate the virtual environment deepens an appreciation for that avatar and its world. Based on the nature of music visualization mentioned above, by adding a sensorial channel through a physical controller, and establishing a user's emotional attachment through aesthetic consequences to the user's control, SONANOS intends to amplify the multisensorial experience.
My Design Goals in SONANOS
My design in SONANOS is focused on letting users express themselves through creating visual artifacts with corresponding musical qualities. Learning the logic of the SONANOS world, by exploration and challenge, is a big part of the experience. SONANOS is not a tool for freeform music composition. Unlike music composition software, SONANOS refactors the process of composing music into an immersive exploration of a multisensorial environment. Boundaries and constraints in this world establish simple yet flexible mechanics to guide and encourage users to create their own content. Aesthetically, the entire experience within SONANOS maintains a consistent visual and musical theme. I chose electronic techno music as its musical theme because the main mechanics, which will be mentioned later, are based on repetitive patterns of a segment of music called a loop. The loop is frequently and effectively used especially in that specific genre of music. Also I chose abstract geometric motif in the visual theme as that is consonant with the musical theme.
My Philosophy of Interactive Art
Interactive art can only be completed by engaging users and accepting their interactions, so it should be flexible enough to adapt to user input. That doesn’t mean the designers should not have their own message or philosophy. The interactive designer’s necessary role is to define the mechanics of how the system operates and those mechanics are the vessels for the designer's philosophy. Instead of expressing their values directly or explicitly, the interactive designer influences the user's experience and the user's understanding through the mechanisms of their interaction and the consequences of their decisions. Interactive art is about letting the user try many alternatives and learn from that exploration. The virtual environment is a safe space where the user can learn without worrying about external consequences. It is where a user is invited to consider alternate perspectives and exercise new behavior.
IV. ABOUT SONANOS
1. SONANOS WALKTHROUGH
From the beginner’s perspective helps we may effectively overview the whole experience. So, here I write explain the system in the order of a beginner’s walkthrough.
Overview
The first thing users are encouraged to do is construct a seed. Once constructed a seed is planted into the ground, so that it may sprout into a tree.
The Seed
A Seed can be generated by combining one visual element (VE) and one musical element (ME) together.
While moving around the 3D world, users encounter various symbolic particles which represent visual elements or musical elements. What kind of symbolic particles you can find is explained in the section titled "Visual and Musical Elements." If the user hovers the mouse cursor on a visual element, a volumetric shape is displayed, and if the user hovers on a musical element, a sound is intoned. The user may temporarily combine a VE and ME together, by clicking on one of the elements, dragging the mouse cursor on top of the other element and releasing the mouse button. After deciding on a desirable combination, the user can link the VE and ME together to construct a seed. In this way, the user can create a personalized connection between each visual and musical element. This seed is then stored in the user's inventory. After creating a seed, the user can plant it into the ground.
The germinated seed grows into a tree, which corresponds to a segment of music called a loop. A loop is often used to create rhythm through periodic repetition of sound. SONANOS uses ambient techno music, because: a loop is one of the frequently used devices in that particular music genre, and a loop conveniently maps a musical structure to a visual pattern, which induces synesthesia.
After planting a seed on the ground, the user needs to collect other MEs or VEs to feed the seed, so that it may grow. Some of these elements are water particles, which are distributed throughout the environment. By dragging and dropping a water particle onto a seed, the user collects that water particles and feeds that seed. Each time the seed is fed water, the seed generates branches. Each generation of the branch is equivalent to one measure in a musical score sheet. So, if a user wants to create an eight-measure loop, then the user should collect eight water particles and feed these to the seed.
At each branch, is the intersection of a parent trunk and set of child stems. At each branch, the user can graft a musical element or visual element to change the corresponding structure of the child stems, and all descendent branches stemming from that intersection. By default, a newborn stem inherit the same musical and visual structure of their parent trunk. By inserting a unique visual element or musical element at the intersection, the user is not only changing that section's musical or visual structure but is also recursively mutating all descendent stems. This technique is especially effective in loop-based techno music, which frequently employs repetition of a musical measure or phrase. Just as interesting music has many variations, the user is encouraged to construct different trees to create various loops. Additionally, after constructing a tree, the user can duplicate that tree anywhere in the virtual environment.
Playing Musical Trees
After constructing several trees, the user can compose a song. To compose music out of trees in SONANOS, the user must understand the temporal mechanics which, in this system, is equivalent a traditional musical time line. As you know music is placed in a one-dimensional time line and the process of musical performance is to follow the linear instructions. In SONANOS, the performance depends on the avatar's location. The avatar can move around the trees placed on the ground and, like a conductor of a symphony, the user can target one of the trees to be the active performer. The user can only choose trees within a limited range from their avatar's position. By default, the closest tree to the avatar, that is also in range, is automatically selected as the active performer. Consequently, if there is no tree in range, then no music will play. The user can select a active performer by clicking the Root of another tree, or may turn off all the music by temporarily disabling the target function itself. The user can think of the avatar as a musical cursor that determines which loop to play. Also, of course SONANOS does not render single-channel music. Instead trees exist in a three-dimensional coordinate system, and the performance of the music depends on how the avatar moves around the trees and targets them. By carefully placing each tree relative to others, the user can conduct a musical tour around the space and control the flow from phrase to phrase.
Layers
So far, we have been playing on the ground of SONANOS environment, but through planting and growing trees, the avatar accumulates experience points. After achieving a threshold, the avatar evolves into a higher-level incarnation, which exposes advanced functionality and maneuvers to traverse the environment.
The vertical dimension of the environment of SONANOS is divided into layers. At the beginning, the avatar cannot jump or fly to reach a higher layer of SONANOS. Three layers exist above the ground level.
Each layer has different musical and visual elements that the user cannot find on the ground. Higher layer elements can construct creatures that actually move around their layer and interact with trees there to generate composite musical and visual effects. This emergent behavior is also an important part of the user experience. When the avatar advances to the final level of evolution, the user can direct any creature and attract each to the ground layer and initiate their interaction with ground objects. Those interactions generate dynamic visual and musical effects. At the final level of evolution, the avatar can also freely maneuver around the environment.
The Angle of Destruction
The Angel of Destruction is ironic, because it absorbs trees and creatures, in order to compose its structure. Therefore, the Angel of Destruction both creates its own structure and destroys that of all other creatures in SONANOS. While the Angel of Destruction proceeds user character also loses its abilities one by one and eventually goes back to the original status. This creature is the antithesis to all other lifeforms. It is analogous to sound waves, which being played through, disappear into thin air. The difference between playing the music through the avatar and through Angel of Destruction is the Angel of Destruction generates different effects while interacting with all other objects. This encourages over the top effects that punctuate a dramatic finale. How the mechanics behind the Angel of Destruction’s interaction with the rest of the world is explained in the section, "Angel of Destruction".
Recording Music and Visualization
At any level of SONANOS, the user can save their progress. Ont top of the data about the structure of the world, the user can also record performance data in a same file. The performance data includes the avatar's position and active performer while interacting with objects in the world, synchronized to the time line. The time limit of a recording is about 3 to 5 minutes. This allows the user to create and record a cinematic experience using camera control.
2. INTERFACE
The design of the interface was influenced by the goal to make the software downloadable and playable by the general public. So I chose mouse and keyboard input devices. By minimizing the buttons and adopting the traditional usage of the mouse and keyboard on most operating systems, the controls are intuitive and broadly accessible.
Combination
In the inventory, the user can combine a visual element and a musical element into a seed by clicking both, and then dragging this combination to a capsule-shaped slot in the Seed Window.
Inventory
Visual elements and musical elements are stored in separate inventory categories. Each of these inventory categories can hold up to eight different slots of items, and each slot can hold up to 4 copies of the same type of item.
3. SONANOS WORLD LAYERS
The environment of SONANOS consists of four layers. The first layer is the ground where users create trees. A tree establishes the basis of the music by using instruments such as percussion and bass. So, the ground layer has visual elements and musical elements that correspond to those instruments. The second and the third are layers have elements that may construct creatures. Creatures differ from trees on the ground level, because creatures employ simple artificial intelligence to around the world and interact with trees. The elements on the second and third layers also correspond to musical instruments that are used to compose melodies. The last level is where Angel of Destruction resides, which is related to the finale.
4. Avatar
The avatar gains experience points and advances levels to unlock abilities to explore the environment.
5. VISUAL AND MUSICAL ELEMENTS
Here is the table about the categories, elements, and usage of visual and musical elements.
1) Visual Elements
| Category | Name | Usage |
|---|---|---|
| Structure | Mesh Shape | Changes shape of branch mesh. |
| Tree Growth Pattern | Decides how the child branch is going to be grown from its parent. | |
| Skin Color | Applies diffuse color of the branch mesh. | |
| Texture | Applies texture of the branch mesh. | |
| Particle | Attaches Particle effect into Intersections. | |
| DefaultBranch Angle (Pitch/Yaw/Scale) | Decides default angles and scales for the child brances growing out of a parent branch. | |
| Animation | Growth(# branch) | Animating by increase/decrease the number of branches of a Tree. |
| Branch Angle (Pitch/Yaw) |
Animating by manipulating each angles of the child brances. | |
| Scale | Animating by changing the scale of branches. | |
| Color | Diffuse color animation. | |
| Particle Properties | Animating by changing particle movement, change of color and shape. |
2) Musical Elements
Musical Elements are categorized by the types of musical instruments such as percussion, bass, melodic instrument, pad and sound effects.
6. CONSTRUCTING TREES
Figure 1 - Performance
7. TREE POSITIONING AND PERFORMANCE STRATEGY
By composing the spatial relationships between trees, through their relative distance or proximity, the user can influence the musical composition. Since the user's distance and targetting commands determine which tree is performing, the user is not composing a linear process of music. Instead the user is constraining the phrases and transitions, while affording degrees of freedom for a future play session to explore an alternate performance. But by recording a performance data file, the user can also create a scripted performance, which includes the session's movement path and resulting music. To play music, it is essential that the user control the avatar’s movement and targeting system to transition between musical phrases. Therein lies the performance aspect of this system.
8. INTERACTION OF CREATURES
Creatures, which are generated in the second and third layer of the environment, each have their own simple artificial intelligence to interact with trees or other creatures. The creatures move around in their layer and are attracted by trees objects or other creatures. Their visual and musical effects synchronize to that of the objects with which they are interacting. While interacting, a creature's visual and musical structure are influenced by the parameters of the other participant, or participants', structure. After the interaction finishes, they revert to their prior structure. The user can attract or repel a creature, which may stimulate it to interact with a proximate tree or creature, or even to temporarily link the head of the creature to the body of another to explicitly direct which tree or creature will interact with it.
9. THE ANGEL OF DESTRUCTION
The Angle of Destruction is an allegorical entity, which suggests a narrative finale to the artistic experience. This creature only comes to life by the user’s choice. By placing this creature in the highest layer of the environment and letting the user voluntarily destroy their art, I intend to induce a cartharsis. There is a fundamental emotional consequence to destroying one's own creation. I wanted to compare this act of ‘destruction’ to artists who bravely throw away what they have done and start fresh to create something innovative and genuine.
10. USER CREATED CONTENT
World Structure
By selecting "Menu - Save," the user can save the SONANOS world they’ve created, including inventory, seeds, trees, creatures, and the current status of the avatar. But that doesn’t include information about yet uncollected elements in the environment.
Performance Data
By selecting "Menu - Record Performance," the user can start recording the interactions in the environment. To protect the computer's processing and memory, the duration is limited to 3 to 5 minutes. The recorded interaction data includes the avatar's position, targeting, camera angle, the user's commands and so on. This can be collectively considered as replay data. Only one performance file can be saved in one save file. To create other performance file, users needs to save the world using a different file name.
11. PERFORMANCE
My interactive visual music (iVM) performance is also using the same core engine, but that software is fundamentally different because it uses a musical keyboard as an input device. So, the keyboardist can fully express him or herself through nuances of stroke duration, attack, and velocity. But the mechanism for in iVM for mapping musical events to visual effects is predetermined prior to the performance. The iVM mapping mechanism is flexible enough to reflect the performer’s improvisational expression of music interpretted visually.
12. TECHNICAL BACKGROUND
The application for procedural graphics is programmed for windows PC using C++ and Direct3D. Also interactive music is programmed based on DirectMusic.
PRIOR ART
Oskar Fischinger and other visual music artists[1]
Researching previous visual music arts, which is mostly non-interactive, has been essential to my project. Without understanding this fascinating art form, augmenting this foundation with interactivity would not have been possible. Oskar Fischinger’s work, including paintings, motion paintings and films, are some of my personal favorites, as well as worthy research material for the study of visual music.
Toshio Iwai - Piano as image media, Light Well, Musical Insects, Electroplankton [2]
Toshio Iwai's work is inspirational because they focus on the user's self-expression.
J. Walt Adamczyk - Autocosm : Gardens of Thuban [3]
This is a live performance at SIGGRAPH 2005. Using a tablet in real-time, J. Walt Adamczyk creates and animates procedural alien-like plants and creatures. By using a joystick from flight simulators, he also dynamically moves the camera to create a live, cinematic experience.
Tetsuya Mizuguchi - Rez [4]
Rez is a primary inspiration. It has the closest visual and musical theme to the environment of SONANOS. Not to mention its synchronization between graphics and music, Rez’s visual style is arguably more abstract than other games. There are many good lessons I leant from it in terms of how interactivity, graphics and music can cooperate to enhance the user experience.
Fumito Ueda - ICO, Shadow of Colossus[5]
ICO is not a music game at all but the feeling of exploring a huge, foggy castle, and the ambience of the game influenced my art direction.
Shadow of Colossus has a vast terrain in which the user explores to find a colossus. Since the terrain is mostly empty and events rarely occur, some users were not entertained, but for me, it was aesthetically pleasing. The journey through the open terrain while riding a horse was a memorable aspect of its gameplay. SONANOS has open space that the player can explore, as opposed to being confined to trails or scrolling through a screen, which is the common fare of music games.
Amplitude[6]
This game has a freestyle mode in which the user can
place notes on the tunnel and compose music with limited controls, but it lacks the feeling of creation.
Guitarhero Freestyle Mode which is not included in actual game but only presented at GDC 2005[7]
At the Experimental Gameplay Workshop GDC 2005, developers from Guitarhero presented their excluded Freestyle Mode of the game. They explained why it was abandoned in the release version of the game. It is difficult to simultaneously let the user create their own music, maintain a fair scoring system, and generate pleasing music from the user's freestyle input.
Keyboard Mania[8]
This game is the same as beat mania, except that the user uses a small MIDI keyboard controller. Using a MIDI keyboard is similar to my performance piece, but this is also not about creating music.
Visualization Tools - WinAmp, Windows Media User [9]
Artists from Advanced Visualization Studio (AVS: WinAmp Visualization Tool, open source)[10]
Visualization tools offer many interesting procedural graphics that can be manipulated in real-time by the parameters of the music. Since this software only analyzes the frequency of the music to manipulate the visual effects, it does not always produce a meaningful or noticeable connection between the music and the graphics.
Demo Scenes [11]
Demo scenes employ cutting edge real-time procedural graphics. There are lots of inspiring works that show how procedural graphics can visualize music effectively.
ACKNOWLEDGEMENTS
This thesis was guided by the advisors Chris Swain, Mar Elepano and Tracy Fullerton. Additional thanks to Ethan Kennerly for encouragement and editing.
[1] Oskar Fischinger and other Visual Music artists http://www.centerforvisualmusic.org/ArtistsPages.html
[2] Toshio Iwai - Piano as image media, Light Wel, Musical Insects, Electro plankton http://en.wikipedia.org/wiki/Toshio_Iwai
[3] J.Walt Adamczyk - Autocosm : Gardens of Thuban http://www.johnadamczyk.com/Thuban.html
[4] Tetsuya Mizuguchi - Rez http://en.wikipedia.org/wiki/Tetsuya_Mizuguchi
[5] Fumito Ueda - ICO, Shadow of Colossus http://en.wikipedia.org/wiki/Fumito_Ueda
[6] Amplitude http://en.wikipedia.org/wiki/Amplitude
[7] Guitarhero Freestyle Mode which is not included in actual game but only presented at GDC 2005 http://en.wikipedia.org/wiki/Guitar_Hero
[8] Keyboard Mania http://en.wikipedia.org/wiki/Keyboard_mania
[9] Visualization Tools - WinAmp, Windows Media PlayerUser http://en.wikipedia.org/wiki/Music_visualization
[10] Artists from Advanced Visualization Studio (AVS: WinAmp Visualization Tool, open source) http://en.wikipedia.org/wiki/Advanced_Visualization_Studio
[11] Demo Scene http://www.scene.org