“Nickel Concert” allows the player to act as a conductor for a virtual audience. The setup makes use of a standard webcam and a physical “clapper”, pictured here:
Using OpenCV’s blob tracking, the program observes the motion of the player’s hands, and based on the position and size of the blobs, different audio is played. (Hands in a higher position translates to higher pitched sounds, etc). Here’s a sample recording of a session: Download file
The Arduino circuit is fairly simple, providing just a direct connection to two servos. The arms and hands of the clapper are attached to the top of the servos, so that controlling the servos in a limited arc range gives the impression of applause:
The musical preferences of the audience are unknown to the player, however, and they have to be discovered through experimentation. At runtime, a “sweet spot” is randomly selected, a combination of both pitch and instrument type. The closer the player is able to play near this sweet spot, the faster the clapper will applaud. Thus, it acts as an analog performance indicator, in an attempt to convey the audience’s “mood.”
The biggest challenge is creating this setup was probably the physical construction of the clapper. Lacking any metal frames to attach the servos to, everything from glue and putty, to paper clips and binder clips was used. It still didn’t end up very stable, unfortunately, preventing the clapper from moving fast enough to give a better simulation of applause.
Another aspect to expand upon, if this project were to be taken further, is the algorithm by which the musical performances are judged. Having a more complex heuristic (one that observes sounds over a duration, instead of just based on the current sound) might lead to more engaging interactivity.
Another angle of the clapper:
Ed attempting to physically interact with the device: