CTIN 534 Experiments in Interactivity I

brad's stereo images

roof of the MOCA

roof of the MOCA in "color"

toys

flower

Anaglyph channels in Photoshop

Here's a quick introduction to one technique for transforming flat images into stereo pairs in Photoshop.

anaglyph.pdf

History of Visualizing Sound, Pre-Computers

My Paper is Here

A Look at the Pepper's Ghost Projection Technique

Pepper's Ghost is a projection technique of sorts as it involves a light source that projects an image on a large piece of glass at a 45 degree angle that "reflects brightly lit objects offstage, superimposing them with the on-stage set and actors."1

The trick is called Pepper's Ghost after John Henry Pepper, a professor at London's Royal Polytechnic Institution, the man who first used it in a stage performance in 1863. Some sources argue that Henry Dircks should receive the credit, since he came up with the idea a year earlier. However, it is noted that Dircks' ghost was only a small-scale prototype, while Pepper's was a full-sized stage show. (As an aside, I think this points to the value of having good content to support a new entertainment technology, rather than just a prototype, because people tend to remember content. It's an important issue to consider as we set out to create our thesis projects).

Pepper's Ghost was, in a sense, a natural evolution of the "magic lantern show," already popular in England at the time, and it went on to be featured in various "ghost shows" (or "bogey shows") at British fairgrounds. Its popularity lasted until the early 1900s, when cinema replaced it as the latest "trick."

Despite its disappearance as a show in itself, the Pepper's Ghost illusion continued to be used in stage productions that required a semi-transparent ghost, including productions of Hamlet, Macbeth and the Christmas Carol. However since the large glass blocked a lot of the sound coming from the stage, the trick was used only sparingly, until the advent of microphone/speaker sound systems.

The effect, and several evolutions of it, is still used today. The Shuftan Process uses an angled, partially mirrored sheet of glass to allow miniatures to be "rear projected" into a scene with an actor. The miniatures appear to be full-size, and reduce the requirements of set-building. The effect is also used in modern haunted houses, including the famous Disney Haunted Mansion, which features the ghostly images of animatronic ghosts dancing in a ballroom. A variation of the Pepper's Ghost technique is also being used in special effects theaters, including the Holavision Theater developed by BRC Imagination Arts.

This theater uses "computer-controlled combination of special effects, theatrical lighting, audio, projection and live performance."

One non-entertainment use of the effect is the heads-up display on modern fighter jets, which allows pilots to see vital information without looking away from the cockpit window.

Sources:
1 http://www.dafe.org/misc/peppers/peppers/htm
2 http://www.brcweb.com/products/special-effects-theaters
3 www.wikipedia.org - "Pepper's Ghost"
4 "Adventures in Cybersound", http://www.acmi.net.au/AIC/PEPPER_BIO.html
5 "Giving Up the Ghost", http://www.thegalloper.com/backstories/0702ghost.html

544 Syllabus

I meant to post this weeks ago but I forgot. Sorry.

CTIN544 Syllabus

real-time compositing for mixed reality

As titled, my paper centers on techniques for mixed reality, and real-time compositing, and concludes with a proposed project. An example of a stage performance that uses these techniques is the Big Art Group out of New York. The stage setup generally consists of a screen with cameras mounted on top that point at performers who are visible from the waist up on a stage behind the screen. Video of the performers is then projected onto the screen. While their show Flicker doesn’t go far beyond this simple (but effective and highly choreographed) setup, House of No More takes it to its logical next step where actors perform in front of green screens, and computer generated imagery is composited into the image.

While the use of green screen can be an integrated aspect of a well-designed show, for my own work I would find it to be too limiting. Visually, the green surface would need to be lit uniformly, and be fairly bright, which would likely turn it into a large and glaring visual focus and could easily become overly distracting. On top of this, the color spectrum of objects and performers would need to be restricted away from green. While this could offer some creative potentials (such as the green shoes used in House of No More), over all it’s too limiting.

Academic research by Paul Debevec at the Institute for Creative Technologies at USC takes another approach utilizing near-infrared illumination in place of green screen technology. Using an IR sensitive camera with an IR pass filter, Debevec was able to identify a fabric that absorbed most visible light (thus appearing black to the eye), while reflected most IR light (appearing white to the IR camera).

This was then used in place of a green screen material behind the performers and lit with IR LED lamps. Two cameras were employed for compositing purposes: one being sensitive to only visible light (and not IR), and the other being sensitive to IR (and visible light) with an IR pass filter placed over the lens. The IR camera sees the IR illuminated background screen as mostly white, while the figure is dark. This grayscale feed is then used to key out the figure in the RGB feed from the visible light camera.

While this helps with the problem of spill (where the green screen reflects green light back onto the performer), the background is far from completely white, and in a more complicated lighting setup, could cause the system to confuse the figure and background. This lack of robustness is likely due to the fact that the system was not designed for real-time, but rather a complicated post-production process of compositing.

A system that has many interesting possibilities is one developed at the BBC’s R&D division. Instead of a green screen, or a black IR reflective material, they use a retro-reflective material that is highly reflective (similar to the function of materials in street signs). The material is hundreds of times more reflective than white paper, and can be lit with a small amount of blue light emanating from around the camera lens.

This material in general is not new and has been used in special effects for some time, however the specific material developed by the BBC in conjunction with Reflecmedia is much more robust than it’s predecessor. Normally, retro-reflective material must be lit and viewed at near 90 degree of incidence. This new material however has a much wider reflectivity angle allowing it to be more flexibly lit and viewed.

In addition, the BBC developed a camera tracking system for orientation and translation that uses a series of circular markers with different printed patterns of concentric circles placed on the ceiling. Termed the Free-d system, the production camera has a second small camera mounted on top of it with a LED lamp pointed at the ceiling. The lamp illuminates the markers, and the camera tracks them to calculate the position and orientation of the larger camera in real-time.

This retro-reflective material and camera tracking system were employed by Industrial Light and Magic (ILM) to create a system for real-time compositing of CGI elements on the set of the movie AI: Artificial Intelligence.

On set Steven Spielberg was able to see a low quality virtual set composited in with the live action actors and set, and thus was able to more precisely control the movement and placement of camera, actors, and sets in relation to the future CGI.

A project idea I have is to line the interior surfaces of a room with Reflectmedia’s special retro-reflective material for use in an augmented reality performance installation, and then light the material with a series of non-visible IR lamps. I would then attach IR sensitive cameras to a conventional augmented reality setup (HMD + visible light cameras) to make it possible to use the IR camera feed to key out the retroreflective screens in the visible light camera feed and composite in real-time graphics into a feed to the HMD screens.

Camera tracking would likely take the form of an inertial tracker with a cord for orientation, and an IR LED on the top of the HMD tracked by a camera in the ceiling for translation (although a setup with ceiling markers would be interesting to attempt with some software engineering help, perhaps from IMSC). The interior would be lit with visible light as well, and allow physical objects to be interacted with on top of a virtual background. Determining the z-depth of real objects to aid in compositing real-time graphics in front and in back of physical objects would be ideal, and perhaps something possible with the Canesta system (to hopefully improve upon previous multi-video camera z-depth work).

Projection Systems for Special Venues & Theme Parks:

The majority of special venue and theme park projection systems are designed to enhance the viewer’s sense of immersion. This is typically done through the use of broader and brighter imagery. Visual impact is further enhanced by quality sound systems that create a “larger than life” sound. Other enhancers include seating movement, ultra low frequency speakers, smoke, smells, lasers, and other physical effects.

Filling the viewers’ field of view can greatly enhance the sense of visual immersion. A human’s maximum limits of color discrimination are approximately 60 degrees vertically and horizontally. This “cone of vision” varies with hue, saturation, light intensity, age of viewer, etc. Field of view can be increased with comfortable head movements of approximately 30 degrees vertically and 45 degrees horizontally (see diagram below).

The most common specialty venue projection system is IMAX®. These specialized theaters are characterized by steeply raked seating and large (up to 100’x74’), and slightly curved, vertical screens. They can seat up to 1000 people and are often attached to Museums to increase membership and ticket sales. Visuals occupy approximately 100% of the viewer’s field of view. These theaters also have a proprietary “rolling loop” vacuum projector system that moves a 70mm horizontal film stock through the gate at 24 frames/sec. The new IMAX®HD system increases the frame rate to 48 frames/sec. for an even more realistic image quality.

IMAX® “Magic Carpet” theaters experimented with a second screen positioned below the audience and viewed through sections of glazed floor. This was most effective for aerial, aerospace, or underwater content. However, the cost of building such theaters was prohibitive. OMNIMAX® theaters use a similar system as IMAX® with a 75’ hemispherical-dome screen. A fisheye projector lens is situated at the center of a 30 degree sloped, 350 seat theater. These are well suited for planetarium and laser shows. Both OMNIMAX® and IMAX® can feature an optional 3D SOLIDO® dual synchronized projection system with IR signal, polarized viewer goggles.

Showscan® is another large format projection system used in special venues like theme parks and World’s Fairs. These theaters tend to be a little smaller than IMAX® but offer a standard 60 fps (frames per second) flickerless high definition image. Showscan screens are a wide aspect ratio, closer to the wide-angle projector, and curved to produce a brighter image than most theaters. Films typically last 20 minutes and many offer Dynamic Motion ® moving seating for synchronized motion simulation.

Showscan also offers a 3-D system utilizing Twin left and right 70mm projectors @ 60 f.p.s. Some say this is the most effective 3-D system on the market. Additional 3-D venues host Omni Films’ ESI-3D® (a good polarized 3D system for standard theater screens) and Stereovision 70 (split 70mm frames off one negative).

Another popular special venue and theme park projection is Disney Circle-Vision 360. This proprietary system capitalizes on a cylindrical 360-degree image. 9 curved screens are placed side by side with synchronized projectors placed at the opposite seams. Content is shot utilizing specially constructed rigs with nine 35mm cameras. Most of these theaters have the audience stand so they are free to turn completely around during the films. Handrails are essential as the slightest tilt in horizon convincingly suggest the entire theater is tilting. Other similar systems include Vision 360, Superscan Cinesystem, Cirkinovision, and Ultra Toruscope 3 and 5 screen systems.

Iwerks Imagine 360 and Swissorama 360 are similar from the audience’s perspective. However, both of these systems have a seamless projection. Shooting 70mm film with a specialized fisheye lens through a doughnut shaped mask provides each image frame. Utilizing a matching fisheye projector, mounted in the center of the round theater, completes the illusion. Although the image is less vibrant that 9 separate projectors, the camera itself is much less clumsy to use than the previous 360 rigs.

Other less common special venue projection systems include Cinema 180 featuring a quadrispherical screen, Motion Master with its hydraulically activated seats, and motion based simulators with standard projectors housed on the moving platform.

Disney’s California Adventure hosts a unique set of theaters in the “Soarin’ Over California” attraction. Two OMNIMAX domes are set on their sides and the viewers are lifted in bench-like seats to a four-tier vertical stack theater. Each bench tilts and pitches in conjunction with the scenic fly-over of California’s greatest landscapes. A grandiose score by Jerry Goldsmith and familiar smells pumped into the theater, like pine and citrus, enhance the epic dangling foot journey.

On a completely different scale, Disneyland’s “Haunted Mansion” showcases several specialty projection systems. These include Flying ghosts projected on three story scrims, singing faces projected on gravestone busts, and a proprietary gypsy’s head fiber optic projection on the interior of a floating crystal ball. These and more Imagineering tricks make the Disneyland’s Haunted Mansion one of the best collections of specialized projection systems.

Sources/Bibliography:

BKSTS - The Moving Image Society (http://www.bksts.com/)
IMAX Corporation (http://www.imax.com)
iWerks Entertainment (http://www.iwerks.com)
Building A Dream – the Art of Disney Architecture by Beth Dunlop
http://www.doombuggies.com/

(I was not able to upload related images to this page)

Motion Tracking for Two using IR and Greyscale

Before filtering and adjusting greyscale.

After!

For our upcoming performance, Brad and I are using a camera with an infrared filter in combination with "infrared" bathroom heat lamps behind the projection screen, so the screen appears white to the camera. We then have costumes with enough separation on the greyscale to distinguish us from each other(Brad is currently represented by a sheet of duvateen.

Links to some Journals & Archives

Digital Performance Archive

Intersections of Art, Technology, Science & Culture - Links

Intelligent Agent

Switch Journal

Ctheory

Idea about my "Two weeks project"

So my project is related to projection. My original idea is using camera mounted behind the wall of IML, or window of ZML to project what's outside the room. I'll adjust to make it looks like a hole or window on the wall. The purpose of this project is to bring in sunshine and communication between the two sides of the barriers.

But I've other idea about a performance piece. I think painting process is somewhat interesting to watch. I'd like to do some 3D projection painting. We can have a sculpture with no texture set up in the room. And I'll paint metal, wood & human skin on top of it to create this illusion of the object. Or we can have a participater make a pose and take a photo of him from the webcam. And I can paint tattoo or body armor onto his/her body. When I'm done, ask the participater to repeat the same pose, and we can see the body armor or tattoo there.

Which one would you like? If we all need to do some kind of performance, I can do the second one. Otherwise I think the first one is more useful.

Broken Stero Pieces For Use

I donated my broken stero - has two cassette decks and a 5-CD changer, none of which work. I also went ahead and brought the two speakers and the remote control that came with it - those still work fine. Hope they come in handy for someone!

color tracking Jitter patch

Here's the patch we built on Monday night, all spiffed up and annotated.

colorTracker.pat