Ethan Kennerly

Perry and I (and maybe you too?) are starting to develop test cases for a Wii Media Lab. Toward that end, here is a proposal for some uses, risks, and plans for the lab. Your comments, especially CRITICAL problems are most welcome. Thanks already to Mark Bolas, Marientina Gotsis, Perry Hoberman, and Lily Cheng, for their advice.

Last semester, I had made models for the panorama presentation of the story bible for my thesis in Google Sketchup.

In the course of making the models, I fell into an infatuation with Google SketchUp, but using a mouse to model was like modeling with a bar of soap. So I began imagining a one-to-one correspondence of physical pointer position with a virtual cursor. My first sketch was for webcams, but then I remembered Ken's post.

Ken Leung posted a link to a novel Wii input device probject, by Johnny Lee. Johnny Chung Lee at Carnegie Mellon University prototyped a low-cost whiteboard and finger-controlled user interface using a Wii mote as the camera.

and so an idea began for a...

Wii Media Lab

Ethan Kennerly Use infrared camera and infrared LED to intuitively control the cursor of a variety of applications while being free from the keyboard and mouse. These are input device systems for the Zemeckis Media Lab. My requirements for these input devices were:

• utilize the room
  • to present interactive media
  • to navigate an virtual environment objects
  • to model objects
• low cost
  • $50 to $100 per station.
• available hardware
  • Wiimote
  • bluetooth port
  • infrared LED
• multipurpose
  • easy to recreate for future users
  • easy to apply to other artists

There are four input setups for the Zemeckis Media Lab. Each one is illustrated in the Google SketchUp model. Please refer to the corresponding scene tabs for each proposal. SketchUp is free.

2D pointer + 6 DOF device

Traditional Wii remote and sensor bar configuration. Application: Collaborate while developing interactive project, such as ZML Spaceship, and intuitively note item on screen. Most any motion mapped in WarioWare Smooth Moves could be programmed by a skilled engineer.

2D multipointer

Wii mote, infared light pen, by Johnny Chung Lee. Application: Control multiple points for scaling or rotating on 2D plane.

2D multitouch whiteboard

Wii mote camera, infrared light pen, by Johnny Chung Lee. Applications: Draw on the wall. Control multiple points.

3D multipoint tracker

2 wii motes, 2 infrared light pens. Applications: Model 3D image. Navigate cursor in 3D environment.

Risks

• Engineers (such as I) are unfamiliar with programming input devices.
  
• Several Bluetooth connections interfere with each other and slow down data transfer.
  
• Cameras can be confused by multiple points of light from neighboring input stations.
  
• User may be confused by six degrees of freedom by moving in 3D space.
  
• User becomes fatigued from maintaining limbs away from body and supporting devices after several seconds.
  
• User may be confused by ubiquitous shape of wii mote.
  
• Adapting device drivers and user interface control maps to software applications may be time consuming for each application.
  

Precautions

To lessen risks, the following precautions are intended:

1. Test Wii remote on a Mac and Windows machine.
2. Test operation of standard Wii system in Zemeckis Media Lab.
3. Test previous implementations of Wii input device for PC that have similar objectives.
4. Scan heat sources in Zemeckis Media Lab. Test range and field of view and field of emission of camera and infrared LED. Direct camera away from heat sources in Zemeckis Media Lab.
5. Minimize number of bluetooth devices in use, by starting with a single wii remote.
6. Employ a real-time introspective development language that I'm familiar with, Python.
7. Start development in an OS that I have access to at school and home and the most experience with: Windows XP.
8. Establish unit tests to quickly confirm operation of camera, wii remote, device drivers, and other units that are developed.
9. Roleplay rehearsal of input device manipulation to detect fatigue, disorientation, confusion, and lack of affordances.
10. Test simple cases of system operation and functionality for usability.
11. Cite adaptation required to modify system to control novel software.

System Architecture

The system employs components and binds them with Python.

Hypothetical stages of signal flow on Windows XP for simple motion test:

1. Raw image in physical space.
   
2. Blobs of infrared light on photocell array of wii remote.
   
3. Two dimensional points within wii remote.
   
4. Bluetooth encoded 2D points from wii remote transmitter.
   
5. Bluetooth encoded 2D points at Bluetooth-USB adapter.
   
6. Bluetooth encoded 2D points in Windows PC.
   
7. Wii 2D points decoded by pyBluez, Python binding to Bluetooth drivers.
   
8. Wii data decoded by wii.py script, adapted from Wiiewer.
   
9. Matrix transformation of 2D input to normalize plane of 2D points.
   
10. Simple gesture recognition, as jestur.py.
    
11. Inject input to mouse movement through windll.py.
    
12. Display device shows mouse pointer moving.
    

Test case requirements

To evaluate the potential, a prototype is suggested as a test case. This test case:

• Renders simple 3D scene.
  
• The input device enables the user to translate, rotate, and scale an object in the environment.
  
• With 15 minutes training, a graduate designer is faster at performing these manipulations with the input device than with a mouse.
  
• The prototype can be developed by a couple full-time graduate designers within their technical knowledge set in a couple weeks.
  
• Prototype costs less than $200 in hardware and software.
  

Here are some test cases. At least one of these will be functionally prototypedduring the experimentation class.

Test case: Chopsticks

This prototype uses Python-Ogre to render a simple 3D scene. Python-Ogre is free, open-source, and interpreted, for real-time editing. PythonCard is used for tool user interface.

Test case: Model manipulator

Based on "Wii input to Blender" at Georgia Tech. Within a 3D modeler, Blender, script control of user interface by Python. Use pitch, yaw, and roll to rotate a model. Use buttons to select editing mode. Translate, rotate, and scale selection by wii remote manipulation.

Test case: Free paint

Paint on 3D environment by wii remote input by generating objects lights and particle systems along path of motion.

Technical Requirements

Minimum System Requirements:

1 GHz Pentium III or AMD Athalon

8x Speed DVD Drive

3 GB Hard Disk Space

64 MB Video Card

DirectX 9 compatible sound & video drivers

Windows XP

Wii remote with 2 charged AA batteries

Infrared light source with power supply

Bluetooth adapter

Software:

Python 2.5

Costs

Each station costs:

• $30 wii remote
• $20? infrared LED with power
• $30 Bluetooth-USB adapter
• Open source software
• 1 hour? installation labor by skilled information technologist
• 1 day? adaptation labor by skilled information technologist
• A day? per month of service for maintenance and support per site

Project costs:

• 3 weeks of part-time research and development for test case by team of two (both are engineers and interaction designers)

Alternatives

Wii remote transmits point data for four points at 100 Hz, or 100 times per second. While an infrared camera may have a comparable price, the frame rate is lower and the image signal is raw, requiring further processing.

Sources

Research suggests that the prerequisite technology exists within the technical requirements, and that the objectives of the proposal are possible by skilled engineers.

Wii pointer, whiteboard, desktop VR, by Johnny Chung Lee.

http://www.cs.cmu.edu/~johnny/projects/wii/

Untested wii remote access exists for Python.

http://www.wiili.org/index.php/Wiiewer

Wiimote communication test.

http://www.wiili.org/index.php/Wiimotecomm

Wiimote input types.

http://www.wiili.org/index.php/GlovePIE

Bluetooth for Python

http://pypi.python.org/pypi/PyBluez/0.9.2

http://code.google.com/p/pybluez/

http://people.csail.mit.edu/albert/bluez-intro/

How to make a webcam work in infrared

http://www.hoagieshouse.com/IR/

Python gesture recognition

http://sourceforge.net/projects/jestur/

http://projects.reichbier.de/gestikk/

Non-wii motion

http://www.youtube.com/watch?v=UGImvHEtUpE

Wiimote accelerometer for Vpython

http://www.youtube.com/watch?v=eZ9xKAFwPLg&NR=1

Mouse control

http://kvance.livejournal.com/985732.html

http://mail.python.org/pipermail/python-list/2003-August/219221.html

http://sourceforge.net/projects/pywinauto/

OSC for media control

http://createdigitalmotion.com/tag/windows/

OSCulator for Mac
Wii input and device drivers

http://www.osculator.net/wiki/Main/HomePage

Mac Python mouse automation?

http://appscript.sourceforge.net/

http://appscript.sourceforge.net/pyosa.html

http://wiki.python.org/moin/MacPython/AppleScript

Wii input to Blender on Windows

http://www.selectparks.net/modules.php?file=article&name=News&sid=628
Some users have had trouble making this Wii-Blender mashup work.

http://blenderartists.org/forum/showthread.php?t=109517

NNiimote (neural-net gesture recognition)

http://jorgecardoso.eu/blog/index.php?/archives/92-NNiimote-update.html

Wii to MIDI on Windows tutorial

http://vjkungfu.tv/archive/wii-to-midi-walkthrough-for-windows/
GlovePIE for Windows

http://carl.kenner.googlepages.com/glovepie

Python OSC

http://wiretap.stetson.edu/downloads/kits/pyKit.tar.gz

http://opensoundcontrol.org/implementation/python-simple-osc

http://www.nabble.com/simple-osc-on-python-to5170289.html

wxPython mouse control

http://www.google.com/search?hl=en&safe=off&rls=com.microsoft%3A*%3AIE-SearchBox&rlz=1I7GFRD&q=wxPython+mouse