Two sketches explaining the use of Arrays and Array-lists

One sketch explaining how to dynamically switch between video streams

Four sketches to interface with the Kinect controller to display depth images, track distances, track hands, and track center of mass

You can download the file here: Processing_code_examples

- Download the zip-file

- Uncompress the zip-file

- Move the resulting directory to the following location:

  /Users/yourName/Documents/Processing/libraries/

For week two please prepare the following excerpt of the first chapter of Mary Ann Doane: The Emergence of Cinematic Time.

You can download the reading materials here: The Representability of Time

Reading Assignment Week 3:

For the third week please prepare the following essay by Henry Jenkins: Game Design as Narrative Architecture.

You can download the reading materials here: Game Design as Narrative Architecture

Reading Assignment Week 4:

For week four please prepare an excerpt from Gene Youngblood: Expanded Cinema.

You can download the reading materials here: The Aesthetic Machine, Computer Films

Reading Assignment Week 5:

For week five please prepare the following essay by Gonzalo Frasca: Simulation versus Narrative.

You can download the reading materials here: Simulation versus Narrative

CTIN501_Week14_ambient_algorithmic_crowd_source

http://www.tulselupernetwork.com/basis.html

http://www.tulseluperjourney.com/

The following is a short recapitulation of what we covered in the meeting.

Computer Vision Experiments

We used the external library OpenCV for these experiments, which you can find here: http://ubaa.net/shared/processing/opencv/

There are two parts that need to be installed, first the Intel OpenCV framework and second the processing library. Please follow the instruction given at the webpage (in case you did not install the libs already in class).

Blob Detection

The first example is a simple implementation of the blob detection function of open cv. In order to facilitate the detection of blobs the camera image is turned into a posterized black and white image. Play with the threshold value in order to adjust it to the lighting situation of your set-up. The best results for image analysis are achieved when you have a uniform and evenly lit background. When you start your application you have to calibrate it by reading the image of the background into memory (command “opencv.remeber()”). The current image of the camera is then compared to the saved image and resulting differences are tracked. In the sketch I am providing for download you can hit the space bar in order to save an image to memory.

The variables minArea and maxArea allow you to specify the minimum and maximum size of the blobs you want to consider. Play with these values to see the effect. The variable maxBlobs determines how many blobs are detected.

Download the sample sketch.

Activity Tracking

The second example is a small modification of the first sketch, which you can use to track the activity in front of the camera. The main differences between the first sketch for blob detection and this one is that the camera image is saved to memory after it is analyzed and used for comparison with the next frame. The strategy is that minimal or no movement means minimal or no difference between between the current image and the previous image, resulting in 0 or a low number of blobs detected. A lot of activity in front of the camera creates a lot of difference between the current and the previous frame, resulting in a higher number of blobs. Play with the size and the amount of blobs to find the best result for your purposes.

Download the sample sketch.

Color Tracking

The same sketch can be modified to track only blobs of a certain color (such as find the red blob in the image). The code is the same as in the first example, at the beginning save a reference image of the background. After the image has been converted to a posterized black and white image and was analyzed for blobs you have to restore the original color image (command “opencv.restore(RGB)”) and analyze the colors in this image in the areas of all the blobs found in the first part. The sketch is sampling an area of 5 by 5 pixels around the center of each blobs and compares the average colors in this area with the color that is searched for. The color to look for is defined with its red, green, and blue components in the variables redSearched, greenSearched, and blueSearched. If a blob is found that corresponds in color to these values it will be highlighted with a green frame and its area will be determined. By tracking size variations you may be able to infer to the distance of the object from the camera. The precision of these values depends very strongly on the set-up of your background, lighting situation etc. The variable searchTolerance allows to specify a certain tolerance within which the values may differ from the original value you are looking for. Play with all these values to find the right values for your situation.

Download the sample sketch.

Face Detection

The last example is a sketch that uses OpenCV’s ability to detect faces in the camera image. Only faces looking into the camera with both eyes visible are recognized by the algorithm. The variable facesAmount will reflect how many faces are currently detected in the camera image. This number allows you to find out how many people are currently facing the camera. Again, play with the values in the sketch to find the best results. For this sketch you do not need to save a reference image to memory. Faces are always detected no matter what background – but sometimes, if you have a very noisy background other patterns in the background that resemble faces might be tracked as well.

Download the sample sketch.

Radiant Historia is a Japanese RPG that, after roughly a six year drought, weaves some of the most successful elements of prior works into its experience.  The music is heavily influenced by the music of Xenogears; the plot is influenced by games in the Ivalice series of games including Vagrant Story, Final Fantasy Tactics and others; the art style is similar to tri-Ace RPGs that had come out before their attempt to switch to 3d, which was not well accepted.  The importance of the influence of these prior works, and the creation of a successful game using those ideas in the modern ecology of game sales, is that, due to systemic factors of the direction the games industry went in the late 90s, these games were killed.

RPGs, requiring at their core, a great deal of dramatic writing, evocative music, and implementation of a compelling battle system, lost out when the industry swung toward next gen consoles that were sold almost entirely on their ability to run high level 3d graphics; RPGs, having to traditionally cut corners with graphics or use unique techniques in order to make games that look good without having to devote all their resources to graphically animating the world of the game, were left in the dust.  Radiant Historia proves that there is still a dedicated market for those types of games; and with the emergence of newer distribution venues such as iPhone and Facebook, the limiter created by next gen consoles is being lifted.  Once the systemic problem — the consoles and their aim — was removed, the drought of RPGs ended almost abruptly.

The meatiest part of Radiant Historia, and the most relevant piece to interactive cinema, is the successful implementation of its interactive narrative structure.  The plot is simple; the country is at war, and the special intelligence branch of the government and the army at at odds for who controls the country’s power — it’s a very Hamlet style of story.  The way the interactivity works is that the player chooses very early whether to side with the special intelligence side or the army side; this main narrative branch is the only major branch in the plot, going against the overuse that killed narrative branching.  After choosing, the player can then switch freely between the events of either timelines, visit the past, and use information only available in one timeline in the other timeline.  Radiant Historia improves upon the interactivity of the branching idea in new ways, in addition to successfully avoiding misusing it: it implements the narrative map itself into the game and uses it extensively to fill specific gameplay elements.

The narrative map — literally the map of all the events, branching plots and structures visually represented in the game — is used as a puzzle space in order to solve problems within the game.  An example is: when working for the special forces, you need to know the intentions of General Hugo, the leader of the army, in order to continue so you, the player, bounce around in the available times searching for an appropriate moment to speak to General Hugo.  Additionally, the map is used to fulfill two common gameplay mechanics: checkpoints, the places where players revive if they die or reach a dead end in the plot; and new game+, the ability to revisit events that have already happened.

Its use of the plot points as checkpoints interweaves the story with the gameplay mechanics in a new and successful way.  This weaving also improves how the story is experienced in different ways.  Traditionally, a game story does not have many of the conveniences of other media that communicates story; conveniences such as being able to fast forward a movie, or skip a chapter that is not interesting or relevant in a book.  However, in Radiant Historia’s model where the plot map is heavily integrated with the game itself, conveniences such as the ability to skip text, fast forward events, and jump around time, makes the presentation of the plot much more open and accessible than most games.  Any chapter is open to revisiting; any chapter can be skipped and returned to later; most events can be avoided or put off, so the player can make meaningful choices about which plot elements to access and in what order.

The interactive cinema project I’d love to share is an interactive trailer for the movie Scott Pilgrim vs. the World, which turns the trailer into a mini arcade video game, and lets viewers interact with the story and uncover an inside look into the movie by acquiring points and achievements in the game. The interactive game trailer seamlessly matches the fantastical graphic novel nature of the movie. It also puts the game console in the viewers’ hands and utilizes the engaging nature and techniques of a video game to absorb the users into the movie/game world in an absolutely fun way.

Scott Pilgrim vs. the World is a fantasy-comedy based on the graphic novel Scott Pilgrim written by Bryan Lee O’Malley. The film is about Scott Pilgrim, a young musician who meets the girl of his dreams, an American girl Ramona. In order to win Ramona, Scott must defeat Ramona’s “seven evil exes” who are all teaming up to kill him. The film itself is a hybrid of movie and video game, with many video game special effects embedded throughout the film to bring out the fantastical and spectacular battles of Scott Pilgrim’s fighting with these evil exes. In these fights, Scott needs to navigate through a sophisticated system of points and levels in order to survive and win. By doing so, the director blurs the distance between reality and fantasy and presents to the viewers a world of wonder yet with a sense of intimacy tapping into familiar feelings viewers had with their own video game experience.

The overall tone of the trailer matches the fantastical graphic novel nature of the movie seamlessly. And while playing, many clues of the movie will pop up on the screen for a limited amount of time, and viewers need to click on them to earn points and find as many pieces of information about the movie as possible. When viewers find a hint, they can check out the links in the interactive trailer’s information stream for more details about the film. The more hints they find, the more points they gain and achievements you unlock.

I love this trailer especially because it breaks the usual pattern of the video game movie that throws the player into the world of the game; it instead brings the game world to the viewer through a cinematic experience, and by clicking on these hints, more behind the scenes information is revealed at that moment, which gives the viewer a richer viewing experience through their own actions. It integrates well with the movie itself, and allows the viewer to experience the same gaming sensations experienced by the main character in the movie. This interactive trailer draws in the viewer and prepares the viewer to dive into a deeper and more immersive movie-viewing experience, and gives the viewer an interactive cinema experience itself.

Experience it yourself:

http://movies.yahoo.com/feature/scott-pilgrim-interactive-trailer.html

I did a Google search for “interactive entertainment”, landed on the website for the Academy of Interactive Arts and Sciences and saw that they had the 14th annual award results (http://www.interactive.org/awards/2011_14th_awards.asp). Scrolling down the list of winners, I noticed a game called “Dance Central”.  Because I am interested in dance, I decided to research it.

Dance Central takes the dancing game into a another dimension by utilizing a peripheral for the Microsoft Xbox called the Kinect.  The Kinect uses an RGB camera, depth sensor and multi-array microphone to allow a “controller-free gaming and entertainment experience”.  A 3D scanner system called Light Coding interprets 3D scene information from a continuously-projected infrared structured light (http://en.wikipedia.org/wiki/Kinect).  So it can tell if you are pointing your arm towards the camera (it does not just rely on silhouettes for positional information).

The technology also allows facial and voice recognition.

The really exciting thing to me was that Dance Central was a dancing game that now involved the whole body, not just the feet, as in Dance Dance Revolution (or just the controller buttons, as in 1st generation dancing games).  Here is a clip of an enthusiastic user dancing to Dance Central: http://www.youtube.com/watch?v=E9MZH1m-RuA.  She has split the screen to show herself actually dancing.

Obviously, this is a highly interactive experience that brings full, 3D body motion to the table where Microsoft, Sony and Nintendo sit (Sony also has the Playstation Eye and Playstation Move; Nintendo has the Wiimote + Nunchuk and the Wii Fit device).  But what about cinema?

As I was watching clips of people playing Dance Central, I noticed that what was playing on their television screen looked very much like a music video–certainly a sub-category of cinema.  Everything from silent movies with musical accompaniment to Thriller or even today’s 3-4 minute music videos contains narrative elements to varying degrees.  The lyrics to Dance Central’s songs almost by definition contain narrative (some more light and repetitive than others, of course).

There was also a point in every song where the user can freestyle; the user can see  him or herself on-screen as a rainbow colored humanoid blob (the Kinect’s view of the user) moving in real-time.  A stuttered clip of the freestyle in actual, real video (not a colored blob) then plays back for the user on-screen before the experience continues.  I found this fascinating because 1) the game reveals to the user how it sees or interprets the user’s body, and 2) it also attempts to allow the user to break through the confining boundaries of the rest of the game play’s pre-defined choreography. I see a parallel between this and the tension between tight narrative control (narrauthor) and relinquishment of narrative control (simauthor).  It changes up the experience, but ultimately I believe they can do much more with the simauthor side of things.

The decision to stutter the freestyle clip struck me as interesting as well. The speed and jerkiness of the stutter made all the freestyle clips I saw look humorous, abrupt and barely coherent. I would guess that they used the stuttering algorithm as a filter to maintain the divide between reality and the game experience; that is, seeing an actual, unfiltered video of one’s freestyle dancing might be embarrassing or uncomfortable to the player, a potentially dangerous feeling similar to breaking the “suspension of disbelief” in traditional cinema (to the game producers) that is mitigated by the drastic revisualizing algorithm that they throw on the clip.

Getting back to the simauthor side of things, I would really like to see the game incorporate the ability to capture the user’s motion via the Kinect and map it onto its avatars. Then a user could choreograph completely original routines and create movements and combinations that may not even be in the game.  Users could then share routines with one another and in that way, teach one another new moves and styles and compete as well.  Providing this added level of interactivity would take this game far beyond what it is and turn it into a rich creative production tool as well as a globally accessible, technologically enhanced dance community.

As it is right now, it exposes users to new movements and rhythms of dance, provides highly active, healthful exercise, and makes dancing fun in a way that turning on the same music and dancing in front of a mirror may not necessarily do.  The elements of hidden characters, advancing to harder levels, gaining points (which you can compare online), cool avatars, and 1 or 2-player play provide attractive features that have made Dance Central one of the top games of the year, and Kinect’s poster child.