OddThinking

One of the many reasons I have been quiet recently is I am working on a personal project – an amateur stop-motion video. The final product will appear to be a large group of my friends playing music in an orchestra. The raw material I have is dozens of sets of still photos of my friends pretending to play instruments in front of a green-screen. Each set has about 10 photos of the person playing the instrument.

I am working on an interesting challenge, and I thought that I might benefit if I wrote it out as a blog article to force me to think it through.

For the time being, I am just going to consider wind and brass instruments (e.g. flute, oboe, clarinet, trumpet, trombone, etc.)

Plan A

The original, original plan was to take a photo of every different legal fingering position for the instrument, and map that across to the particular notes being played in the music. Whenever a B♭ was played, the photo corresponding to the B♭ fingering would be substituted in.

This original, original plan is almost feasible for an instrument such as a trumpet. If you assume that the lip-position is not discernable, then there are only 8 fingerings – several notes correspond to the same fingering. It would take some research, but is doable.

However, I looked up the fingering charts for an oboe and found there were (from memory) 44 different positions. That would take too much studio time, too much categorisation time and be too boring for my volunteers.

Plan B

So, the original, original plan was replaced with Plan B before the photo shoot. Plan B involved taking approximately 10 photos of each instrument being played. Where convenient, these would be associated with their correct notes. Where a note didn’t have a corresponding photo available, the software would assign a random fingering to that position, and then maintain that association throughout the piece. A musician who played that instrument might notice the discrepancy, but hopefully the casual viewer wouldn’t notice the “anachronism”. Perhaps some smarts might be added to prevent two sequentially-played but differently-pitched notes from having the same fingering, so that some movement occured between notes, but that would probably not be required.

The Shoot

Associated with the shoot was my birthday party.

You can view the causality either way: in order to get volunteers to come to the photo shoot, I invited them to a party in a pub near the studio, or in order to get my friends to come to my party, I invited them to a photo-shoot in a studio near the pub.

I relied on my friends to take up roles such as Assistant Director, Photographer, Wranglers and Tour Guides. They were terrific, and I am ever so thankful for the work that they did. They made the day run very smoothly, and I was – several times – banished from the studio and set to the pub because they had everything under control.

But as part of having many short shifts of volunteers, and volunteers training other volunteers (while the director lazed in the pub) is that there was a certain element of Chinese Whispers.

In particular, the “franticness” of the playing slowly increased over the day. At the beginning, I have photos of flautists keeping their heads and bodies still, while just moving their fingers. By the end of the shoot, I have saxophonists wildly boogeying left and right as they play.

Now, this isn’t quite in keeping with the sedate orchestral image I was originally planning, but I can adapt. More importantly, it offers great challenges for the stop-motion – I can’t have the player swinging their hips 90 degrees with each note played, if they are playing, say, 200 notes per minute – they will look like they are holding an out-of-control jack-hammer to their lips.

I want to be quite clear that I am not blaming the volunteers involved. I could have made my needs clearer, I could have kept a closer eye on the proceedings and, really, this problem is a pretty minor one compared with what could have gone wrong. I am overall still pleased and thankful.

(Some of the production people claim that they tried to get some serious playing photos before some more fun ones. That is certainly true for some instruments. For other ones, I really only have a couple of “sedate” playing images, making the animation rather dull if I don’t incorporate some of the later images.)

Plan C

So today I came up with another idea.

Suppose we ignore the fingering – no-one will notice the fingering positions of a dancing saxophonist. Instead, just focus on the instrument/body position of the player.

For each set of photos of a single player, we can compare each pair of photos and rate them as say “tiny movement”, “moderate movement”, “large movement”.

You know what – this will be easier to explain if I can degenerate to Comp Sci geek…

Let each of the 10-20 photos be a node in a fully-connected bi-directional graph, and each edge be weighted by the size of the delta between the positions.

Idea #1: Discard all the edges which are rated higher than “moderate”. Then, each time a different note is played, one of the connections is randomly chosen to allow the player to continually make small, but random movements as they move through the space. Hopefully the random big jumps will be turned into more gradual hip swinging.

Idea #2: Reduce the probability of leaving a node along the same edge it entered through, so the player doesn’t “vibrate” as much – jumping back and forth between the same two nodes. Note, some nodes will only have one other node similar to them, making vibration inevitable.

Idea #3: Increase the probability of leaving a node along the same edge it entered through if the previous note is repeated. e.g. let the player vibrate if they are playing A-B-A-B-A.

Idea #4: Tag each edge with the next note played during each transition, and then re-use that transition if you find yourself doing the same note transition, so the player is likely to (but not definitely) repeat their movements when they play the same refrain a second time.

Idea #5: Don’t discard the large weightings. Instead just make them stochastically less common. This will reduce the chance that a player will become randomly stuck in some small set of nodes, vibrating until the piece is over. It will increase the chance that they do get stuck in such a situation for short periods.

Idea #6: Occasional frantic movements may be justified and natural. Could there be some trigger that makes larger movements more likely for some transitions? Big jumps in tone? Jumps in tempo, key or volume? Semibreves?

Categorisation

With a large number of instruments and a large number of photos of each, coming up with a rating for each pair is likely to be laborious.

Richard A. suggested the other day that I farm out some of the categorisation work. I guess I could make a web-site that displayed two merged photos and asked for a personal rating of the differences, and then asked some volunteers to walk through them. Would writing the web-site (Oh gawd! PHP?) be worth it? How much consistency will I get through different people rating them?

Could I do it automatically, by taking a delta between the two images (at the pixel-level) and then somehow summing the result to get a metric for how much the picture had changed? Would that metric be reasonably correlated to a manual rating?

Conclusion

Writing this down has clarified it in my mind, and triggered a few more ideas.

It has also revealed it to be a lot of work – and possibly the result will be unwatchable. Hmmmm….

On the other hand, the solution (at least a manully-rated one) is likely to be useful for string instruments too (violin, cello, double-bass, etc.) – ensuring the bow positions transition rather than jump.

Suggestions very welcome.