efn ‘at’ usc ‘dot’ edu
Interactive Media Division - USC School of Cinematic Arts
February 27, 2007
Sonorous is an experiment in game design which aims to explore the possibilities of using music and sound as an alternative to traditional story-like narrative as the primary means of creating context for gameplay, and facilitating player engagement. Sonorous takes the form of a shortened Real Time Strategy game, where each unit the player controls takes on the properties of a musical instrument in the player’s own temporary musical ensemble group. Through player choices and actions, the units will dynamically create and alter the musical score. The primary experiential goal of the piece is to evoke a grooving, or joyful dancing reaction in the player. The primary research goal of the project is to identify several tenets of Interactive Sound Design as they relate to games, in the hope of unveiling new game mechanics and systems which could be applied to arcade style and casual video games.
Keywords:
Sonorous, Interactive Music, Musical RTS, Game Audio Synchronization, Game Event Synchronization, Music/Narrative, Interactive Sound Design, Grooving RTS
Main Body:
Flow, by Mihaly CsikszentMihalyi, is considered a leading resource on positivist psychology, and a must-read book for game designers. In it, Csikszentmihalyi discusses the "Flow" experience, which is characterized by a clear focusing of the mind, a "harmonious ordering of consciousness", losing track of time, and a deep sense of enjoyment. These experiences were reported to him by people from all walks of life, during a variety of activities, from mountain climbing to working on an assembly line. The description of the experience itself seems quite kindred to what the video game industry often refers to as "Immersion," and it is achieving this state in the player that is primary goal of any game designer. Csikszentmihalyi asserts that the foundation of the Flow experience is becoming engaged in an activity which requires skill, and has clear goals and feedback. Using these criteria, we can discuss the effective merits of different game designs.
Ever improving computer hardware and graphical technologies have encouraged the modern game industry to adopt a model that favors filmic qualities in games. Games have rushed to adopt filmic conventions in cinematography and storytelling, and increasing numbers of film industry veterans are being hired by game studios to help make their games more like film. As such, games have become far dramatic in recent years and the sophistication of in-game stories has increased significantly, and continues to be the subject of much active research. Additionally, traditional film narrative follows a nice tension curve which could easily be mapped to gameplay intensity and difficulty. (i.e. ramping difficulty towards climax of story)
However, some weaknesses in this film-centric approach to game design are becoming evident. Filmic approaches to storytelling are inherently non-interactive, having been inherited from a passive media. In games, this tends to manifest in the form of the non interactive "cut-scene" which disengages a player’s controls while a movie-like animation plays out. When these cut scenes are interspersed between long periods of gameplay, they can interrupt the flow of control the player is used to, and break immersion. Also, games produced with Film-like stories tend to be much longer than an equivalent film would be, since they need to separate the story content with gameplay segments. One of the primary compulsions of these types of games is supposed to be to play through the story, which means that a player must complete all the content. As games get longer and longer, and the average age of the game player increases, many are finding that they simply don't have time for their hobby anymore.
But the primary problem is that currently, film-like story simply doesn't correspond very strongly to skills, goals, and feedback. In a non-interactive form, story-viewing requires almost no skill from the player. Certainly most game stories do not. Most game stories are based on action movies, which is appropriate because most modern games involve the same type of action typically found in these movies, but Action movies are known to be fairly mindless repetitions of the same themes to be broken up by action sequences. This is fine for the passive experience of film, which a person might watch to relax, but it's not the optimal condition for flow. Film-like narrative can be used to provide goals for the player, in fact, this is its most common application - "You're the hero, save the princess/world." The main problem is that these elements are typically too far apart, and rarely adapt to the player's performance, most of the time, the goals are more clearly understood by the implicit goals of the gameplay genre, such as "keep progressing the right," or "defeat all the enemies and survive." These implicit goals probably contribute more to the player's moment to moment experience than the main plot points of the game. But a secondary problem is that story-presented goals are very rarely intrinsic - the player doesn't have to care about the story. If a player doesn't like a character, then a story element where the player has to save that character or interact with them won't be very satisfying. The cut-scene "rewards" which are offered at major plot points could be considered a form of feedback that the player is completing the game. But, as we stated above, these cut-scenes happen very infrequently - a player might have to make hundreds or thousands of small decisions between each cut scene, so its contribution to the overall flow experience is minimal.
Sonorous aims to address these issues by focusing on using musical elements as the primary communicative element from the game system to the player, instead of story elements. Music tends to encourage participatory models more readily than a film experience. The image of a film experience is people sitting together in a darkened theatre staring at a screen. People dance at clubs, groove to their iPods, and sing in the shower. Playing a musical instrument is a common experience, and is more easily related to than many of the situations presented by a game story. Music follows a similar tension curve to dramatic narrative, but that curve tends to operate much faster, and be more flexible than the filmic curve. Musical tension is based on the tenets of music theory - chord resolution, rhythmic changes, etc. Musical changes in tension, however, tend to be sampled by the audience on a second to second basis, as opposed to minute to minute, or less often, when passively absorbing a storyline in a video game. As such, it should be possible to use music to continually reinforce the player's actions in the system, and if we can find ways to incorporate the game mechanics, we have a feedback system that is ripe for producing flow.
Prior Art:
Halo/Ikaruga – Traditional Games with Dynamic soundtracks
Both of these games are traditional action games which dynamically use their soundtracks to cinematic effect. Halo employs a Dynamic Music system which layers in instrumental parts during combat encounters. As the player advances through the encounter, additional instrumental tracks layer in, making the soundtrack more complicated. The end result is a seemingly dynamic action movie score, which augments the action nicely. Ikaruga is an auto scrolling shooting game. Because the levels are auto-scrolling, the developers could manually synchronize the soundtrack to the level, knowing that the player would be in roughly the same situation at each point in time. The developers use this to good advantage, making sure that the music is always appropriate by context, and synchronizing certain in game events to musical cues (notably the engine "boost" of the player's ship shooting into a new phase of the level)
Guitar Hero/DDR – Beat-matching games- designed for grooving and performance
Guitar Hero and Dance Dance revolution are industry leading action rhythm games. Both games are in the "beat matching" genre of game mechanics, in that the primary player challenge is in interpreting an abstract timed sequence of "beats" and performing it in time with the music. Both games have achieved widespread success, and have inspired many other games, creating an entire subgenre. It is also interesting to note that Dance Dance Revolution is one of the first video games to include a "Diet Mode," where the game would track caloric expenditure of people playing the game. This demonstrates potential for video games to be used to motivate people to do other usually unpleasant tasks. Dance Dance Revolution has a static, mostly J-Pop soundtrack, while Guitar Hero layers several guitar streams together based on how well the player is performing. Because of the metaphor, DDR tends to be able to support a wider variety of musical genres in the game, but the experience of Guitar Hero is much closer to actually performing.
Fig 1 - Musical Creatures in Electroplankton
Electroplankton - Interactive musical toys
Toshio Iwai is a world renowned Media Artist from Japan who focuses on Interactive Musical toys, with a particular interest in composition. His most public work is the Nintendo DS title Electroplankton, in which the player uses the stylus and touch screen to interact with an aquarium of musical Plankton creatures, creating a unique musical performance each time. Electroplankton served up a unique and pleasurable experience for the DS, but ultimately failed to win great audience approval. The primary criticisms levied against the title are that it lacked any clear goals, and so became quickly boring, and that it lacked a save feature which would have allowed players to save their favorite performances.
Fig 2
Rez/Meteos/Lumines
Tetsuya Mizuguchi has designed and produced several musical games during his career in the game industry. Of particular note is Rez, which is primarily an on-rails shooting game with abstract visuals and a completely dynamic soundtrack. The soundtrack starts simple, with a simple beat pattern in the background that the player avatar "pulses" to. As the player interacts with the environment, the sound samples generated by their actions tend to fill in the music, which layers and becomes more colorful as the level progresses. Rez has achieved a large cult status in the game industry, and is seen as a largely underappreciated classic. Meteos and Lumines are puzzle games which borrow similar musical ideas from Rez.
Design Overview of Sonorous:
Sonorous is designed as a limited, shortened, Real Time Strategy experience. The player is put in control of a small (one to five members) "army," where each unit is a musical instrument in what is effectively a small musical ensemble. Depending on what actions they are performing, whether standing around, moving towards an objective, or fighting, each unit beats out a musical tune, and together, they play a song. Unit fighting and other actions are synchronized to the beat of the music, causing the entire game to feel like an abstract dance. As in a traditional RTS, player control fundamentally consists of using the mouse for selecting units and issuing orders, although Sonorous's musical elements introduce some action based mechanics and increase the pace of the game. There are two primary ways to win, by destroying your opponent, or by capturing the citadel in the center of the map.
Fig 3 - Concept art for "Lead" Object
One of the defining mechanics of Sonorous is maintaining the "Lead." Each player has a resource known as the "Lead", which is represented by a type of floating energy ball. The Lead is an energy source which can be attached to various objects in the game. Each side only gets a single one, similar to a flag in a capture the flag game, and it can only be carried by a single entity at a time, either one of the player's units, or a building node on the map. Most of the time, the Lead is in the possession of one of the player's units, who effectively is the ensemble's "Leader," similar to the "Hero Unit" concept of a traditional RTS. This unit will gain a variety of gameplay related statistical boosts, such as increased offense, defense, or speed, and will also play the lead part of the ensemble's song.
Unlike a traditional RTS, however, the player may at any time transfer the Lead to a different unit, who will then become the Leader. The new unit will gain the associated stat boosts, and the old unit will return to his default state. Also, the new unit will start playing the lead track, and the old unit will move to a support track. In this way, the player can dynamically re-arrange the song the units are playing.
Fig 4
The Lead can also be used to capture territory on the map. Players gain new units by capturing buildings which they discover as they move about the map. The capture process for a building involves transferring your team's Lead to the building you're trying to capture. The Lead will slowly awaken the building and capture it. This can be a dangerous process, because while the Lead is contained within the building, you don't have access to its unit-strengthening properties.
The player can also earn bonuses for swapping the Lead between units at the appropriate times, adding a subtle beat-matching mechanic to the game. The Lead has a "charge" meter which fills as the unit who holds the Lead fights and accomplishes tasks. The fastest way to charge the Lead, however, is to transfer it between units with the appropriate timing to the song. When the Lead is fully charged, it can be used to activate special unit powers and capture buildings faster.
Basic Progression of the Game
Each player (of 2) starts the game in control of a single unit (which holds the lead). At this point, the music in the game is extremely simple, only the single lead track is playing. Within a short exploration of the starting point of the player's unit, the player should be able to find a node, which they then can capture by transferring their lead from the starting unit to the node and waiting for a short time. After the time has passed, the node will come to life and return the lead back out to the player, and a new unit will spawn for the player to command. The player can now swap the lead between his new unit and the original starting unit.
A player may issue commands to fight the other player's units. When this happens, the units will line up and fire at each other like in a traditional RTS, with the addition that attacks will be synchronized to the music. During battles, players are encouraged to swap the lead between their units often, so that the units can share its stat-boosting capabilities. However, if the unit holding the Lead is destroyed, then the lead itself will become available for attack by the other player. If a player's lead is destroyed, that player loses the game. A player may recapture their lead and re-protect it by moving another friendly unit over it. That unit will then become the lead unit. When a unit perishes, it fades out and becomes a ghost unit, which can still move, but doesn’t emit any sound, and can’t attack. This unit may be resurrected by taking the unit to a player controlled node, re-depositing the lead in that node, and waiting for a period of time. The unit will respawn, and the node will return the lead.
Play continues as the players move about the map capturing nodes, which spawn units. Once all nodes are captured (there will be 7-9 on the starting map), the "citadel," a special node in the center, will appear. Whichever player captures the citadel first wins the game. It is expected that each game should take about 8 minutes or so.
Fig 5
- How Sonorous relates to Feedback and Flow
Sonorous attempts to demonstrate new ways in which games can use music to achieve flow. While the musical systems in Sonorous are meant to be mostly related to feedback (as in the case of Rez), the game is designed such that the player is regularly interacting with the soundtrack. The player interacts with the soundtrack in three ways, by earning new units, by transferring the lead, and issuing orders and actions to units. All of these techniques are tightly integrated in the game's mechanics, which adds the skills and goals elements of flow. The player should be interacting with the soundtrack every second or so, and the system should be continuously responding with rewarding feedback.
- Art Design
Sonorous employs an abstract art style, meant to look pleasing to the player, but to focus their attention more on the gameplay and sound they are experiencing, rather than a concrete narrative. The idea is for player to think of their units more as interesting toys, than life forms to be empathized with. The building-like nodes and citadel take on blocky structures, as do units which represent percussive instruments such as drums. These will have machine-like, occasionally piston-like movements, while other instruments with smoother, mellower sounds (like flutes) take on a more organic appearance, and animate in wavy patterns. The units will have animations for beating with the music, as well as 3 states for idle, moving, and attacking.
Fig 6
Units will fire bright energy projectiles, some of them seeking, like the homing lasers in Panzer Dragoon. The shots are also synchronized to the music, and primarily move by interpolating across splines. This will be explained further below.
Technical Considerations:
Sonorous is developed using an updated version of the "Bushido" Engine - student developed and used over the last three years by the University of Southern California Interactive Media Division. See Dyadin (http://www.dyadin.com), and Cloud (http://www.thatcloudgame.com), for previous examples of the engine's use. Upgrades to the engine include CG shader support, support for blendshape animation, texture animation, spline interpolation, and a new MIDI-driven dynamic music system built on top of the FMOD sound system already supported in Bushido.
- Streamed vs. Sequenced Music
Although streamed (wave form based, like MP3) music tends to sound better to most players, previous experiments with sound synchronization have revealed several advantages to sequenced music (Note form based, like MIDI) representations over streamed music representations. The first major one is far simpler beat detection. In a sequenced music format, note and beat information is provided directly by the format. This information allows the system to synchronize events in the game to events in the music. In a streamed format, beat synchronization either has to be done on the fly, usually using tricky and expensive FFT-based techniques, or needs to be pre-done by manually adding tags to a wave of music, in a program such as Sound Forge. This technique in the past has proven to be arduous and quite error prone, as often up to thousands of tags are needed with millisecond accurate timing.
Fig 7 - Sony Sound Forge - Inserting Meta Tags in Waveform sound
The second major advantage of a sequenced format is that it's easier to directly manipulate the run-time performance of a sequenced song, than a streamed song. For example, changing the playback tempo of a sequenced song is a simple matter, simply speed up the update rate through the note data, but keep the note data the same. In a streamed system, we need to perform expensive Digital Signal Processing-based techniques to accomplish the same result. Simple solutions, such as doubling the playback frequency of the song, tend to have unexpected and often undesirable results, such like raising the pitch of everything an octave. Also, the layering of instrumental "parts" in and out of a streamed data format requires the streams to be split up into several parts. This can cause runtime memory and CPU usage to balloon, as all the parts are brought into play.
- Musical Architecture
The musical content in Sonorous is stored in a multi-track MIDI file. Each song is composed of a number of "Unit Parts." Each Unit Part is represented by an instrumental music track, with 3 variations (one for each of a Unit’s Attack/Move/Idle states). As the player issues orders to the units, the units switch between playing these variations. There is also a special unit part which represents the "Lead" part. Whichever unit has the Lead will switch to this Unit Part, and the other units will all play their own parts.
During gameplay, only a small subsection of the total number of parts are actually playing. Each unit can only be in a single state at once (so only a single variation is playing), and for most of the gameplay, there will only be a few units on the board. This is how Sonorous maintains its musical uniqueness from playthrough to playthrough.
- Interpolated vs. Physically Based Motion
Much of the motion in Sonorous is based on interpolated motion along splines, rather than a traditional game-like physics model. There are a few advantages to this, but most of them have to do with better synchronization between musical elements and gameplay.
In a parametric system, motion is achieved by interpolating between two known points. This contrasts with a physics-based model, where you know your starting point and your velocity, but not your ending point. Typically, games favor the physics model, since it tends to lead to more realistic results, and it doesn't require any knowledge of what's going to happen in the future. In Sonorous, however, we do have a bit of knowledge about what's going to happen in the future, by looking ahead in the musical data. If we can make an approximation about how long we want an event to take, we can look ahead in the musical data by that amount of time, and then try to find a beat we'd like to synchronize to. This can be done by seeing if the beat is within a certain distance of the time we'd like to occur. If a beat exists in that window, then we can say that we'd like the event to happen at the same time as the beat, and then adjust our interpolations appropriately.
For example, let's say that a unit is fighting another unit, and is going to fire a projectile at him. We know that the projectile will start at the location of our unit (our first parametric point). We know he's going to fire, and let's say we use a dice roll to decide whether or not he's going to hit his opponent at the moment he fires. If he hits his opponent, then we use the opponent as the second parametric point. If he misses, then we choose a point on the ground near his opponent. Once this is done, we can look at the distance between the points to approximate how long the projectile will take to travel the distance. Then, we look ahead at the musical data to see if there's going to be a beat that we would like to synchronize to at that time. If there is, then we can "fudge" the interpolation to make sure that the projectile reaches the second point on the beat. When this happens, we could play the sound of the projectile hitting a little louder, or with a slightly more percussive sample so that the player realizes that his unit is firing with the music.
Future Work:
There are a few major enhancements to Sonorous’s design that will likely not be completed by the project’s public showing in May 2007.
There is room for a lot of work in making the soundtrack more dynamic and interactive, especially with the game mechanics. In its current form, the soundtrack in Sonorous takes a fairly passive role, mostly responding to the player. Mechanics which reward the player for taking actions at the appropriate time are kept to a minimum, mostly for the cost of generating additional code and content. It would be nice if the soundtrack built itself in a less linear fashion, for example, than Sonorous currently allows. If for example, the melodies and contents of the soundtrack were to change more dynamically based on where the player is in the game.
There are a couple more opportunities for player personalization which could also be quite entertaining. It would be nice if the system recorded which tracks were actually played during a round of battle, and offered to export the track in MIDI or .MOD format. This could allow players to replay their favorite playthroughs of the game, and encourage players to be more creative in their play styles, looking at the game as more of an opportunity for performance, than simply a competition with the other player. MIDI and MOD files are commonly playable in most modern media players, so the files could also be easily shared.
We also like the idea of players being able to create their own instrument packages for the units to use. This could entail a player designing the sound a unit makes by say singing into a microphone, and potentially creating an army of him and his friends, which could result in a battle which sounded like an aCapella performance.
Finally, we really like the idea of incorporating controller functionality for the Nintendo Wii Controller. The controller seems ideal for Sonorous, for a few reasons. The pointer functionality of the remote seems ideally suited for typically mouse controlled games, such as an RTS. The motion tracking of the nun-chuck attachment could be used to add a nice drumming-style beat matching mechanic. But perhaps most simply, the speaker on the controller seems ideally suited for playing back the tune played by the selected unit, or perhaps the lead unit.
Conclusion
Sonorous is a game designed to heighten the player's immersion by providing a system of continuous feedback which ties directly to a player's goals and skills. It borrows heavily from the design methodology of other musical games, but tries to take a step in a new direction by incorporating the personalization and indirectness of a real time strategy game. The result is a shortened, but intensified experience which players will hopefully enjoy more for the dynamic experience of a playthrough, rather than a static, traditional story experience.
Acknowledgements
Erik Loyer, Michael Crowley, Perry Hoberman, Mark Bolas, Michael Naimark, Chris Swain, Tracy Fullerton, Peter Brinson, Robert Layton, Vincent Diamante, Todd Furmanski, Jenova Chen, Yi-Meh Hu, Glenn Fiedler, Brett Paterson , Brad Newman, Mike Rossmassler, Martin Middleton, Jen Stein, Marientina Gotsis, Scott Fisher, IMD Class of 2007
References
http://www.iasig.org/ Retrieved Jan 24, 2007.
Bungie Studios, Halo (2001). DVD-ROM (XBox). United States: Microsoft Corp.
Collins, Karen (2007). “An Introduction to the Participatory and Non-Linear Aspects of Video Games Audio.” Eds. Stan Hawkins and John Richardson .
Essays on Sound and Vision. Helsinki : Helsinki University Press. (Forthcoming) http://www.gamessound.com/interactive.pdf. Retrieved February 13, 2007
Csikszentmihalyi, Mihaly (1990). Flow: The Psychology of Optimal Experience. New York: Harper and Row.
iNiS (2006). Elite Beat Agents (Nintendo DS Cart). Japan: Nintendo.
Iwai, Toshio (2005). ElectroPlankton (Nintendo DS Cart). Japan: Nintendo of Japan - (http://electroplankton.nintendods.com/flash.html)
Firelight Technologies (2006). FMOD Ex 4.04. http://www.fmod.org/ Retrieved September 24, 2006.
Glatt, Jeff. “Standard Midi File (SMF) Format”. http://www.borg.com/~jglatt/tech/midifile.htm. Retrieved Jan 10, 2007
Harmonix Music Systems (2001), FreQuency. DVD-ROM (Playstation 2) United States: Sony Computer Entertainment Inc.
Harmonix Music Systems (2005), Guitar Hero. DVD-ROM (Playstation 2). United States: Red Octane.
Konami Bemani Series (1998), Dance Dance Revolution. Arcade. Japan: Konami of Japan.
Loyer, Erik (1999). http://www.marrowmonkey.com/other/resisting.html
Midi Manufacturers Association. Retrieved Feb 02, 2007. http://www.midi.org/
Sony Pictures Digital Inc (2003), Sony Sound Forge 7.0. DVD-ROM. United States: Sony Corp.
Treasure Co. Ltd, Ikaruga (2001). Sega NAOMI Arcade Board. Japan: Sega.
United Game Artists, (2001). Rez. GD-ROM (Sega Dreamcast) Japan: Sega of Japan - (http://www.sonicteam.com/rez/)
Vib-Ribbon (2000). CD_ROM (Playstation) Sony Computer Entertainment Inc. - (http://www.vib-ribbon.com/)
Whitmore, Guy (2003). “Design With Music In Mind: A Guide to Adaptive Audio for Game Designers”. http://www.gamasutra.com/resource_guide/20030528/whitmore_02.shtml Retrieved Nov 23, 2006.
Young, Rob. (1996) The MIDI Files. Prentice Hall.