Webcam Piano with Quartz Composer 3.0
A test in motion detection in Quartz Composer 3.0.
The music is all generated in real-time by me waving my fingers, hands and arms around (or in fact any motion) in front of a standard web-cam. No post-processing was done on the audio or the video.
The concept is by no means new, but still fun nevertheless - and I'm quite happy with this implementation. I'm using a very simple frame difference technique and generating midi notes based on where-ever there is movement (actually, as QC3 cannot send midi notes I had to send the data as OSC and use OSCulator to forward them as midi).
I set up a few scales so only the notes in the chosen specific scale would trigger, and in this video I demo some: chromatic (all the notes on a piano), diminished (for a nice tense feeling), pentatonic (for a nice bluesy vibe), and Zirguleli Hicaz (a turkish scale which is by far my favorite - comes in around 1:33, if you get bored at the beginning skip to that bit).
What I like about this example is that there is no special hardware or user configuration needed - its just a standard webcam looking at a standard me with no special clothing or anything. Anyone with a webcam can just load the software and start playing. Of course a more specific version (e.g. IR Camera filming someone wearing gloves with reflective finger tips) would definitely provide much more control, and I think I will try that someday too.
I'm quite happy with the results at the moment, but will probably try an optical flow algorithm for the next version as well. Not sure if I will stick to QC for that - while its really nice and easy to get something good-looking up and running quite quickly, things can get quite messy for more complex projects. I think the source file for this project pretty much sums up the term 'spaghetti code'. I just am not a big fan of node-based programming, I like to see what I'm doing as text line by line, but we'll see.
I've attached the QTZ file which reads the incoming camera feed, analyzes it and sends out note info as OSC. Also attached, is an OSCulator document which forwards the OSC data to midi. You can then direct the midi to whatever you want.
If you do anything cool with these files please let me know, I'd be keen to see/hear :P
| Attachment | Size |
|---|---|
| Webcam Piano.qtz | 28.76 KB |
| webcam piano.oscd | 3.66 KB |
Delicious
Digg
StumbleUponComments
Navigation
Installations & Performances
Tags
- ActionScript 3.0 (7)
- Audio / Music (9)
- Computer Vision (8)
- Flash (9)
- Game (2)
- GLSL (4)
- Interactive (16)
- iPhone (2)
- Mac OSX (17)
- Maths (8)
- Midi (5)
- Multi-touch (3)
- Off-topic (3)
- OpenFrameworks (5)
- OSC (10)
- Papervision3D (1)
- Particles (17)
- Processing.org (12)
- Quartz Composer (17)
- Sample / Source (22)
- Tutorial / Tip (4)
- VDMX (7)
- Wiimote (3)
Recent posts
Recent comments
- Motorcycle Parts
- Motorcycle parts
- Anti Aging Skin Care
- dance music online
- Cybertiger
- flash video
- gtoledo3@gmail.com
greg:
Hi Memo,
Awesome work!
I took the liberty to show it while doing sets/ labs at the Apple Expo in Paris last week.
Working with a VJ showing Modul8 and a Vixid (and Ableton not being there)
I decided to show 2 technologies: your Webcam Piano and Mrmr (for iPhone/ipod people, being the Apple Expo!) all into Live.
I credited you off course.
I'll post some vids soon (there're in HD!).
Thank you for inspiring me!
best best
greg
memo:
cool. lemme know how it goes!
Anonymous:
awsome.
qc and midi . great.
I want to ask you if it is ok to use this script in an art install. if so how can I credit you.
email me at my email.
Geoff Palin:
Dude, this is amazing, Ive been trying to do similar work in flash using Papervision3D particles. To be honest, I don't even know how to run the files provided?, but it would sure be an incredible flash application. Great work, keep it up, always love the experimental stuff, especially with the interaction!
memo:
Hi Geoff, you can see the a slightly more detailed explaination below (my response to moongolds post). I am thinking of doing a version in Flash as well, but lack of proper Midi and OSC support is putting me off. I know there are hacks to get them working but I've not tried them yet...
moongold:
Hi memo, I'm feeling old and daft at the moment because when I first read about this creation of yours I got it up and running in moments with vdmx and Live. Wonderful.
Now for some reason I can't for the life of me get it working.
Is the Webcam Piano.qtz used as an fx in vdmx or imported into the media bin? I have a regular block about this, if there is any, distinction for using quartz files in vdmx generally.
Do I need to use the osc output in vdmx to talk to osculator, I don't think I did last time?
Does it matter what I launch first, vdmx, osculator, Live?
Thanks for the patch. Get back when you have a moment.
regards.
memo:
Hi moongold, for this example I didn't use VDMX (though you can if you want to, to process visuals even more. I am going to post a sample of that soon).
the process is:
- open the OSCD with OSCulator and keep that running (to map the OSC data to midi).
- open the QTZ with Quartz Composer and run it in that.
- run any midi app to generate the notes.
If you don't have Quartz Composer, I've done a version for VDMX as well which I'll post soon (for the VDMX version just remove the live video input patch for the QTZ and publish the input so you can use it as a qcFX on a layer in VDMX)...
moongold:
Hi memo, big thanks for your expanded explanation :-)
I once again got the webpiano working a treat in vdmx and playing Live sounds. Wonderful and magical.
I am now incorporating some very subtle sounds triggered using your patch for the latest install of my piece 'The Back of my Head' which is showing as part the Derby Festival Fringe in the UK in Sept.
The piece was originally silent but is getting a make-over courtesy of memo.
I will post links up for the new install when I have them.
You can see the original piece here:
http://moongold.me.uk/the-back-of-my-head/
PS
There's a Live instrument, Guitar/Glissando, which I have tricking away in the background triggered by my head/body movements whilst typing, as I use the laptop; it is strangely comforting and is possibly helping me concentrate better!
moongold
memo:
cool, looks good.. and looking forward to the updated videos.
P.S. I sometimes leave the webcam piano on, with some soft spacey piano sounds in the background. There is something strangely therapeutic to just go about your daily activities like working, drinking, eating etc. with it on!
moongold:
thanks memo :-)
I've got this running on an iBook G4 for the installation, it works, ish, frame rate in QC shows as around .60 to .80FPS, yes, point6 of a frame!!
Along with Live and OsCulator nothing else running. OSX 10.5.4
Is there a simple alteration I could make in QC to make it a little more responsive, it comes out at a delay of somewhere 1.2 seconds plus. It probably won't really affect the piece which is calm and contemplative anyway but I have been giving it a try.
'Scale Down Orig' looks a possibility?
Or even drop the colour out completely?
Maybe drop the quality of the iSight input hmm
Or a less intensive midi app... hmm that might work.
I'm thinking as I type. If you get a moment mate, big thanks.
Gareth
Ilan Kriger:
Hello mate,
I couldn't make it work!!
I tryed to put the the webcam piano.qtz at quartz composer.. but it give these error message
> Iterator
Cannot create node of class "QCImagePixel" and identifier "(null)"
> Iterator
Cannot create node of class "QCOSCBroadcaster" and identifier "(null)"
> Iterator
Cannot create node of class "QCImagePixel" and identifier "(null)"
> Iterator
Cannot create connection from ["outputRed" @ "ImagePixel_1"] to ["inputValue" @ "Math_3"]
> Iterator
Cannot create connection from ["output" @ "Splitter_2"] to ["inputImage" @ "ImagePixel_2"]
> Iterator
Cannot create connection from ["noteInfo" @ "JavaScript_2"] to ["qcmcnote" @ "OSCBroadcaster_1"]
> Iterator
Cannot create connection from ["pixelY" @ "JavaScript_1"] to ["inputPixelY" @ "ImagePixel_1"]
> Iterator
Cannot create connection from ["outputIndex" @ "IteratorVariables_1"] to ["pitch" @ "JavaScript_2"]
> Iterator
Cannot create connection from ["outputCount" @ "IteratorVariables_1"] to ["iterations" @ "JavaScript_2"]
> Iterator
Cannot create connection from ["outputGreen" @ "ImagePixel_2"] to ["input2" @ "ColorFromComponents_rgb_1"]
> Iterator
Cannot create connection from ["outputIndex" @ "IteratorVariables_1"] to ["currentIndex" @ "JavaScript_1"]
> Iterator
Cannot create connection from ["pixelY" @ "JavaScript_1"] to ["inputPixelY" @ "ImagePixel_2"]
> Iterator
Cannot create connection from ["outputHeight" @ "ScreenInfo_1"] to ["targetHeight" @ "JavaScript_1"]
> Iterator
Cannot create connection from ["output" @ "Splitter_8"] to ["_enable" @ "OSCBroadcaster_1"]
> Iterator
Cannot create connection from ["normX" @ "JavaScript_1"] to ["inputX" @ "Cube_1"]
> Iterator
Cannot create connection from ["normY" @ "JavaScript_1"] to ["inputY" @ "Cube_1"]
> Iterator
Cannot create connection from ["normX" @ "JavaScript_1"] to ["inputX" @ "Cube_2"]
> Iterator
Cannot create connection from ["output" @ "Splitter_4"] to ["numX" @ "JavaScript_1"]
> Iterator
Cannot create connection from ["pixelX" @ "JavaScript_1"] to ["inputPixelX" @ "ImagePixel_1"]
> Iterator
Cannot create connection from ["outputRed" @ "ImagePixel_1"] to ["inputValue1" @ "Conditional_1"]
> Iterator
Cannot create connection from ["outputResult" @ "Conditional_1"] to ["play" @ "JavaScript_2"]
> Iterator
Cannot create connection from ["outputRed" @ "ImagePixel_1"] to ["velocity" @ "JavaScript_2"]
> Iterator
Cannot create connection from ["outputRed" @ "ImagePixel_1"] to ["inputValue" @ "Math_1"]
> Iterator
Cannot create connection from ["outputRed" @ "ImagePixel_2"] to ["input1" @ "ColorFromComponents_rgb_1"]
> Iterator
Cannot create connection from ["outputBlue" @ "ImagePixel_2"] to ["input3" @ "ColorFromComponents_rgb_1"]
> Iterator
Cannot create connection from ["output" @ "Splitter_3"] to ["numY" @ "JavaScript_1"]
> Iterator
Cannot create connection from ["pixelX" @ "JavaScript_1"] to ["inputPixelX" @ "ImagePixel_2"]
> Iterator
Cannot create connection from ["output" @ "Splitter_1"] to ["inputImage" @ "ImagePixel_1"]
> Iterator
Cannot create connection from ["normY" @ "JavaScript_1"] to ["inputY" @ "Cube_2"]
> Iterator
Cannot create connection from ["outputCount" @ "IteratorVariables_1"] to ["iterations" @ "JavaScript_1"]
> Iterator
Cannot create connection from ["outputWidth" @ "ScreenInfo_1"] to ["targetWidth" @ "JavaScript_1"]
> Iterator
Cannot publish input port ["scale" @ "JavaScript_2"]
> Iterator
Cannot publish input port ["bottomNote" @ "JavaScript_2"]
> Iterator
Cannot publish input port ["velMult" @ "JavaScript_2"]
> Iterator
Cannot publish input port ["imageScaler" @ "JavaScript_1"]
> Macro Patch
State restoration failed on node "Iterator_1"
> Macro Patch
Cannot create node of class "QCImageResize" and identifier "(null)"
> Difference Last Two Frames
Cannot create node of class "QCQueue" and identifier "(null)"
> Difference Last Two Frames
Cannot create connection from ["outputImage" @ "ColorControls_1"] to ["inputValue" @ "Queue_1"]
> Difference Last Two Frames
Cannot create connection from ["outputQueue" @ "Queue_1"] to ["inputStructure" @ "StructureMember_index_1"]
> Difference Last Two Frames
Cannot create connection from ["outputQueue" @ "Queue_1"] to ["inputStructure" @ "StructureMember_index_2"]
> Difference Last Two Frames
Cannot create connection from ["output" @ "Splitter_1"] to ["inputSize" @ "Queue_1"]
> Macro Patch
State restoration failed on node "Patch_1"
> Macro Patch
Cannot create node of class "QCImageResize" and identifier "(null)"
> Macro Patch
Cannot create connection from ["outputImage" @ "ImageResize_3"] to ["Orig_Image" @ "Iterator_1"]
> Macro Patch
Cannot create connection from ["output" @ "Splitter_18"] to ["Music_Scale" @ "Iterator_1"]
> Macro Patch
Cannot create connection from ["outputImage_proxy_1" @ "Patch_1"] to ["inputImage" @ "ImageResize_3"]
> Macro Patch
Cannot create connection from ["output" @ "Splitter_19"] to ["imageScaler" @ "Iterator_1"]
> Macro Patch
Cannot create connection from ["outputValue" @ "Math_3"] to ["inputHeight" @ "ImageResize_1"]
> Macro Patch
Cannot create connection from ["outputValue" @ "Math_2"] to ["inputWidth" @ "ImageResize_3"]
> Macro Patch
Cannot create connection from ["output" @ "Splitter_2"] to ["inputImage" @ "ImageResize_1"]
> Macro Patch
Cannot create connection from ["outputValue" @ "Math_2"] to ["inputWidth" @ "ImageResize_1"]
> Macro Patch
Cannot create connection from ["outputValue" @ "Math_3"] to ["inputHeight" @ "ImageResize_3"]
> Macro Patch
Cannot create connection from ["output" @ "Splitter_7"] to ["velMult" @ "Iterator_1"]
> Macro Patch
Cannot create connection from ["output" @ "Splitter_1"] to ["bottomNote" @ "Iterator_1"]
> Macro Patch
Cannot create connection from ["outputImage" @ "ImageResize_3"] to ["diffImage" @ "Iterator_1"]
> Macro Patch
Cannot publish output port ["outputImage" @ "ImageResize_1"]
> Macro Patch
Cannot publish output port ["outputImage" @ "ImageResize_3"]
> (null)
State restoration failed on
thanks in advance
memo:
Hi, are you sure you are using Quartz Composer 3? (i.e. OSX 10.5 Leopard?). The errors look like they come from QC2 (tiger)...
initi:
beautyfull!!!!!
Amsonx:
hey Meno , i've just tried your work with Numerology .. i'ts great
thanks for share this
a|x:
Very nice!
lux:
this work great with abletonLive+VDMX!