This is an experiment I've been working on for automating the 3D computer animation of musical instruments. I created a Softimage ICE tree that builds a point cloud keyboard animated using a texture map image of a midi track. The music was created using Propellerhead Reason and then the midi track was made into images with a series of screen captures from the sequencer window. After a bit of work in Photoshop the re-assembled midi track image was animated in After Effects and exported as a targa sequence. The resulting sequence is ported to a texture map on a grid in Softimage and the ICE tree reads the data and uses it to animate the keys.
The tune used in this example is Desafinado by Antonio Carlos Jobim
Here's a look at the main ICE tree
First I built the keyboard with a interpolated array node. Because computers love creating an index starting with the number zero I have added an 89th point(pianos only have 88 keys). This will make it easier to setup the shapes later when I'm using point ID's.The extra point could also be used to animate a sustain pedal, pitch bend lever or whatever wave twister moog knobs you might have.
The keys are shape instances.
Some ='s, if's and or's check the value of the point ID and assign the proper shape for each key.
For the black keys I manually created a list using an or node. Not the most elegant example of if node usage, but since there was only 36 of them I manually built the list. Perhaps there is a more efficient way to do this by nesting if's and just leaving these for the "else" branch.
For the white keys, which occur at regular intervals, I made a compound that builds the list based on a single integer input.
Then I just duplicated it and changed the shape and input integer for each key. The chain of add nodes are all set to 12 (octave interval).
A "get closest location" node lets me get the color data from a grid's texture map property.
Then ICE converts the color data to scalar data which drives the particle orientation.
The midi track image used was 708 pixels wide, which seemed like the minimum width possible for ICE to accurately differentiate between all 88(+1 empty zero lane) keys.
The image height is based on the length of the song and frame rate.
If you look carefully you will notice the triplets played in the higher octave during the B section aren't working fast enough.
Next time I'm going to bump up the frame rate and image height for a higher key tracking resolution.
360 video / unity accelerometer / augmented reality
Unity3D app built into final installation / unity accelerometer / augmented reality
