CTIN 534

« Projection Systems for Special Venues & Theme Parks: | Main | 544 Syllabus »

February 19, 2005

real-time compositing for mixed reality

As titled, my paper centers on techniques for mixed reality, and real-time compositing, and concludes with a proposed project. An example of a stage performance that uses these techniques is the Big Art Group out of New York. The stage setup generally consists of a screen with cameras mounted on top that point at performers who are visible from the waist up on a stage behind the screen. Video of the performers is then projected onto the screen. While their show Flicker doesn’t go far beyond this simple (but effective and highly choreographed) setup, House of No More takes it to its logical next step where actors perform in front of green screens, and computer generated imagery is composited into the image.

While the use of green screen can be an integrated aspect of a well-designed show, for my own work I would find it to be too limiting. Visually, the green surface would need to be lit uniformly, and be fairly bright, which would likely turn it into a large and glaring visual focus and could easily become overly distracting. On top of this, the color spectrum of objects and performers would need to be restricted away from green. While this could offer some creative potentials (such as the green shoes used in House of No More), over all it’s too limiting.

Academic research by Paul Debevec at the Institute for Creative Technologies at USC takes another approach utilizing near-infrared illumination in place of green screen technology. Using an IR sensitive camera with an IR pass filter, Debevec was able to identify a fabric that absorbed most visible light (thus appearing black to the eye), while reflected most IR light (appearing white to the IR camera).

This was then used in place of a green screen material behind the performers and lit with IR LED lamps. Two cameras were employed for compositing purposes: one being sensitive to only visible light (and not IR), and the other being sensitive to IR (and visible light) with an IR pass filter placed over the lens. The IR camera sees the IR illuminated background screen as mostly white, while the figure is dark. This grayscale feed is then used to key out the figure in the RGB feed from the visible light camera.

While this helps with the problem of spill (where the green screen reflects green light back onto the performer), the background is far from completely white, and in a more complicated lighting setup, could cause the system to confuse the figure and background. This lack of robustness is likely due to the fact that the system was not designed for real-time, but rather a complicated post-production process of compositing.

A system that has many interesting possibilities is one developed at the BBC’s R&D division. Instead of a green screen, or a black IR reflective material, they use a retro-reflective material that is highly reflective (similar to the function of materials in street signs). The material is hundreds of times more reflective than white paper, and can be lit with a small amount of blue light emanating from around the camera lens.

This material in general is not new and has been used in special effects for some time, however the specific material developed by the BBC in conjunction with Reflecmedia is much more robust than it’s predecessor. Normally, retro-reflective material must be lit and viewed at near 90 degree of incidence. This new material however has a much wider reflectivity angle allowing it to be more flexibly lit and viewed.

In addition, the BBC developed a camera tracking system for orientation and translation that uses a series of circular markers with different printed patterns of concentric circles placed on the ceiling. Termed the Free-d system, the production camera has a second small camera mounted on top of it with a LED lamp pointed at the ceiling. The lamp illuminates the markers, and the camera tracks them to calculate the position and orientation of the larger camera in real-time.

This retro-reflective material and camera tracking system were employed by Industrial Light and Magic (ILM) to create a system for real-time compositing of CGI elements on the set of the movie AI: Artificial Intelligence.

On set Steven Spielberg was able to see a low quality virtual set composited in with the live action actors and set, and thus was able to more precisely control the movement and placement of camera, actors, and sets in relation to the future CGI.

A project idea I have is to line the interior surfaces of a room with Reflectmedia’s special retro-reflective material for use in an augmented reality performance installation, and then light the material with a series of non-visible IR lamps. I would then attach IR sensitive cameras to a conventional augmented reality setup (HMD + visible light cameras) to make it possible to use the IR camera feed to key out the retroreflective screens in the visible light camera feed and composite in real-time graphics into a feed to the HMD screens.

Camera tracking would likely take the form of an inertial tracker with a cord for orientation, and an IR LED on the top of the HMD tracked by a camera in the ceiling for translation (although a setup with ceiling markers would be interesting to attempt with some software engineering help, perhaps from IMSC). The interior would be lit with visible light as well, and allow physical objects to be interacted with on top of a virtual background. Determining the z-depth of real objects to aid in compositing real-time graphics in front and in back of physical objects would be ideal, and perhaps something possible with the Canesta system (to hopefully improve upon previous multi-video camera z-depth work).

Posted by brad at February 19, 2005 09:40 PM

Comments

Posted by: brad at February 24, 2005 01:32 PM

Post a comment