www.multicamnarrative.com
I. Concept and Introduction
There are two typical solutions for the recorded motion image: computer graphic effects (e.g. the pixel effects by Photoshop filters) and branching storyline. Each of them creates two types of totally different image culture. The former mostly cares about what’s happening to the pixels’ RGBA features, their relation and behavior as it calculates the effects for each of them equally. For the value system of computer, the pinky pixels on the handsome young man’s face just equal to those of a piece of shit. The latter evaluates images to the stand of literary structuring. The former easily resorts to the SFX with interactivity hardly with communication of the time, space and action inside of the image. After all, it is for the recorded motion that we are expecting for interactivity. Even as SFX manipulates recorded image, its cinematic narrative function goes zero. And for the latter, multiple ending is inevitable. Obviously there is interactivity at the point of making choice to command which direction the story goes. However, it is in essence at the story layer taking advantage of how sci-fi story – not necessarily cinema – puts things: there’s possibility people go back to the past by time machine to make another choice! We know the interactivity is limited because that the “branching story” doesn’t goes deeper to the perceptual layer of the media to solve problems from deep inside. Great cinematic narrative works well when it perfectly combines two well-woven layers of illusion, the perceptual to the story and vice versa. Abstract thoughts, usually phenomenally reflecting the ideology, come last (Fig. A).
Perceptual Built (visual/aural) < > Action, Events and Story < > Thoughts (Ideology)(Fig. A)
The pause where it branches when viewers are urged to make choice is in fact literally structured. Same structuring is observed when the respective approach to reveal “wide view” are put to comparison: One simple wide shot is good enough for cinema whereas literature has to suspend the time to introduce one by one the items inside of the view in order to accumulate a sense of wide view, and then go back to where it pauses to deliver the actions. We see the different time/space structure between cinema, the recorded media for motion and literature. And, interactivity happens only when the narrative reaches at the fork of the road. Its deducible utmost is: interactive with the “objects inside” the image. This can be taken as inferior copy of videogame, one of the forms of computational interactive animation whose image is generated realtime at request of input. Moreover, in terms of time and space, the branching structure for videogame play is more like the one on the left of Fig. B than the right. The branched time exists only spiritually, not actually can be seen or experienced, even in cinema.
(Fig. B)
Because by nature cinema as the recorded media prescribes that once the motion is recorded on the film or tape, it cannot be changed, otherwise, instead, it’d better to communicate via other media e.g. live performance theatre. Omnipotent medium simply doesn’t exist. Questionable direction leads to less productive interactivity and this might be the reason why the interactive storytelling doesn’t go very far along those two previous directions.
The recorded motion reflects how the matter moves in the space for a period of time, aesthetically, timed space (or spatialized time). Deeper thoughts about the original make of cinema in function of storytelling are helpful. The essence of cinema is to document what happened, real or fictional. For the latter, the recorded media are at a position to document what’s fancifully happened. So, once the fictional story happened and was recorded – “seen/heard” by camera/recorder, it won’t be able to branch and all we have is footage shot by shot inside of which are the fragmented motion. Actually, all we can do with the footage we have to interactivity is to play with CUTs and find out how to arrange them so as to establish combinations of illusive, “believable” time/space in the way both compelling and economic. Since editing is the key factor to determine the architecture of timed space / spatialized time – namely narrative that leads to storytelling, the interactivity could happen if the privilege of putting things together is given to the audience. GOOD interactive narrative could be achieved with strategically organized control over the footage delivery. Where we draw back one step to serve as the collector with data of necessity and economy ready for the interactive narrative, the interactivity for the viewer(s) goes deeper. Classical cinematic narratives do have intersection with interactivity.
(Fig. C)
Adaptations of classical film narrative begin from 3-shot (Fig. C) method to open narrative. The great efficiency of 3-shot method narrative has largely profited from the consistent clarity of spatiality as things go on, of which advantage are taken for the adaptations into interactivity. The deficiency of the 3-shot method is that its spatial closeness cuts off the connection between the subject and the environment it relies on. Fortunately, MultiCam/MultiPOV method (Fig. D) allows the space as open as possible.
(Fig. D)
II. Prior Arts
A. Michael: S.F. navigation Project etc. – Spatiality B. Masha Kinder: Labyrinth Project – Ambassador Hotel C. Time Code: Multiple screen D. Dancer in the Dark: by Lars von Trier with MultiCam (up to 100 cameras shooting simultaneously but edited in the normal way)
III. Project Description
A. Installation Setup:
The installation is set up for one key viewer a time to play with her/his several companies. It’s meant to be like a home theatre environment (Fig. E).
(Fig. E)
It is a standalone installation for one viewer to interact with the movie. On opposite to the multi screen e.g. Time Code, only one image a time will be on the display. The viewer always gets the full storytelling by switching among the buttons just as s/he learns from the beginning of the movie. If the viewer doesn’t switch, the movie still goes on with the default and without any pause.
B. Console design (draft):
(Fig. F)
The console concept is the reflection of the interactive/cinematic reality: there are “free”, diverse combinations of time/space/event to tell a story not just imaginary – it’s not conceptual, it is happening in front of the viewers. Given good manipulation from scripting, storyboarding to editing and installation, the interactive storytelling would be delivered both cinematically economic and visually/aurally/tactually immersive. The metaphor of the console is that: in philosophical measure, although only the most meaningful moments are worthy of memory that later becomes the main narrative of the story history, nonsense trivia have their positions so each viewer could see it in differently way. In cinematic media layer, viewers get OmniView on the story with easy access to different layers of narrative way much open than conventional cinema.
(Fig. G-1, G-2, G-3)
The buttons are in hierarchy accordingly. The cinematic space that attaches to one of the big buttons comprises that (cinematic space) of the three nearest medium-sized buttons (Fig. G-1). And the same thing is between the medium and its immediately nearby small ones (Fig. G-1). Since MultiCam/MultiPOV conducts the narrative, the buttons will be allocated equally to either three cameras of three POVs (Fig. G-2, to be elaborated in following section). So, the buttons on the console are systematically mapped to the time/space/events shot with MultiCam or from MultiPOV (Fig. G-3). It’s worthy of mentioning that since by nature MultiCam makes 360° view possible, panoramic expression can be topologically realized – by switching. The buttons are coated with translucent silicone and lit faintly from inside. Once the viewers lose the track of feeling buttons during the movie, the light helps them to re-orientate. So, in addition to feeling the size, the viewer can easily locate the big/medium/small buttons in the darkness by light. The style design of the buttons plays an active role to communicate the cinematic content as well as deliver the metaphor. Depending on the story, round buttons might feel plain, cubic, straight and strong, and the hare, alternative fun (Fig. H).
(Fig. H)
C. Narrative Mechanic:
The console interface in front of the viewer(s) is right the substantialized topology of the time/space structure of the cinematic reality on the display. The MultiCam and its development to MultiPOV narrative take advantages of it in two ways:
(Seeing the console deeply in terms of narrative structure)
(Fig. I)
Firstly, for a single event with unity to the time/space when/where it happens, e.g. conversation, fights or some spectacles worthy of depictions, cameras depict it from different angles simultaneously. Three main buttons represent three main POVs on this event. Each POV covers part of the space where the event happens. Because the time/space/event associated with buttons of each of the three sections are well assorted (Fig. I), while each shot compensates for shots of other cameras, the active viewer gets the information sufficient enough for understanding how the story goes on without losing the sense of geography and continuity of time. Second, seeing from the disunity nature of time/space/event for cinema, the recorded media, the interactive MultiCam narrative reincarnates itself into the interactive multi-POVs narrative. And this eventually leads to multi-timeline structure: when events are in fact related to each other other than happening at the same time at the same location – each event being pre-edited taking one of the timelines, they can be strategically put paralleled for switching. However, technically challengingly, the pre-edits must be carried out with greatest considerations to its paralleled relatives – moment by moment or even frame by frame – to keep the perceptual continuity at the switch, so as to keep the story going well with perceptual smoothness (brightness/color/sound match). So that it could be always rewarding for the viewers to switch the image among the timelines realtime – either getting multiple information and/or “driving” the story on. Based on this design, the events happening in the timelines can either be in different locations at the same time, or events of same location happening in different period of time – flashforward / present / flashback, or pushing even further, mixing them all as long as the mixture offers on-going narrative or meaningful relation.
The mechanics of the main narrative “drive” is engineered by switching among the three big buttons that are respectively connected to one of the multi-POVs; by pushing the medium-sized buttons around each of the big buttons, the viewer gets a further depiction of the POV, for example, several other compelling side shots, close-ups or slow-motioned equivalents for the main shot, etc.; The small buttons rimming the console are associated with even more expressive or trivial shots. Not necessarily narrative-purposed, the contents associated are prepared to either create atmosphere, details the surroundings or, if necessary, clarify the time or spatiality etc.
While sometimes the viewer(s) switch from bigger buttons to smaller ones within a section to cover the content of different narrative layers, big (buttons) ßà medium ßà small thus macro (narrative) ßà micro (blue interchanging arrows in Fig. I), they also switch in between sections (green arrows in Fig. I). This doesn’t matter much to the perceptual layer because the images can be well designed before shooting to avoid the inconsonance or jarring after editing. Hence the visual/aural images on the display would always be continuous. Sound is in sync with the switches biggerßàsmaller buttons within each section. In case the main “drive” is “event shot with MultiCam simultaneously”, the big button switch gets the sync and match sound. So, the cinematic time/space continuity is guaranteed and the illusion secured for the viewers.
D. Play:
The play allows several other friends by the viewer’s side. Because the interactive narrative is cinematically controlled behind the scenes, so long as the viewer keep switching among the main buttons, for them the storyline and narrative on the display will always be fine – as fine as they’re seeing a conventional movie in theatre or from TV at home. They may request on-the-spot the viewer of which button to push and when to. More interactivity among the group viewers would be cautiously considered as to resort to a voting system. The problems remain are: how to pacify the voters for their non-rewarding votes yet keep them live with the rest of the story till the end? What would the ideal size of the viewer body be with interactivity?
Standalone or ingroup, the urgence to interactive goes stronger when the viewers are more sensually involved. In accordance to the geography rendered by the visual images associated with the pushbuttons, surround sound is set up to enhance the immersion. Moreover, surround sound plays a great part in the narrative just as it does to the conventional cinema storytelling.
E. Further steps:
Further steps would set on adapting the stereoscopic image. Accordingly, lighting and editing concepts would have to negotiate with the real Z-depth. Backlight won’t be as much necessary as the conventional lighting to separate the subject from the background because the “background” changes to back field. And if the key light were kept as strong as it was, the highlighted area would look too much thus mess up with the motivation. On the side of editing, because the cornerstone of editing rhetoric is largely based on the screen equivalence, the previous matching mechanic would most probably turn out inconsonant as the depth match is missing. Other unknown however important issues remain till enough stereoscopic visuals are experienced.
The continuity of visual/aural images requires great editing and cinematography strategies that eventually fall into practical application of visual/aural psychology. For example, in terms of vector and vector field for the images on projection, convenient answers can be found to questions like: How to increase the visual strain by choosing location (with all its visual components) and by framing with dolly or simply by editing? How to match the visuals dynamically for the switches by making use of sample color and sample brightness (Fig. J: snapshot from The Chosen http://interactive.usc.edu/members/yuechuan/archives/2006/11/the_chosen_mult.html) Etc.
(Fig. J Color Map Brightness Map)
IV. Problems/Potential/Foreseeing
Two-hand Console / Group Interactivity / Ergonomics / Dynamic FPS / Console, Console
Besides as an entertaining, storytelling installation, the combination (cinematic illusion, viewer(s)’s spiritual world and the console in between) has great potential to be adapted for the psychoanalysts to conduct narrative therapy.
V. Acknowledgements:
Academic Committees: Andreas Kratky, Perry Hobberman, Michael Naimark, Teddy Chow Jen Stein, Scott Fisher who did great job so that I can shoot the film Crews – my greatest comrades-in-arms Classmates – for your so kind tolerance of my long days’ shooting at 555/2276 Friends and Supporters
Future Cinema, the cinematic imaginary after Film – Jeffrey Shaw and Peter Weibel
Photoplay – A Psychological Study, Hugo Munsterberg;
Film as Art – Rudolf Anheim;
Towards True Cinema – Slavko Vorkapich
Visual Story – Bruce Block;
Illusion – Teddy Chow (aka Zhou Chuanji, 周传基 http://www.zhouchuanji.com/resume/resumeen.htm);
Sight, Sound and Motion – Herbert Zettl