Performance Video Project – Problems Solved!

So there were a variety of problems that came up when I tried to put the animations into the augmented reality Flash files.

Firstly, the animations would appear with a black rectangular background instead of a transparent one, which looks sort of bad because it makes the mushrooms look more like a video than a character that you’re holding, which isn’t the desired effect. So, I asked Matt how he managed to retain the transparency of his videos and found that it was as easy as ticking an “Encode Alpha Channel” box when converting the movs to flvs.

Then the main problem that then caused many other problems was that the video/sound files would play even before the marker was held up and wouldn’t stop playing when the markers were put away. All the marker managed to do at this point was make the video visible and invisible depending on whether or not it was held up. I tried my best to find out how to get the video to stop and start at the right time, but the code was too complex to figure out. So, I spoke to Adam, as he has very extensive knowledge of Flash and he understood what was going on with the code so he told me that my easiest option was to import the sound separately into each Flash file and just use some much simpler code to tell the sound to stop and start at the right times. Thus, I rendered out versions of the animations that were transparent with no sound and then added sound to each Flash file and with Adams help, managed to get the marker to trigger the sound and video at the right times!

However, we did encounter a few errors along the way, such as, for some reason when we first imported the sound, the initial code we used just made the sound loop every frame, so it played the track twenty five times per second! So, after much experimentation we managed to get it to work, which was very satisfying!

So after figuring out how to get it to work perfectly on one Flash file, I then applied that to the seven other Flash files, resulting in eight Flash files; each with a mushroom animation in!

The video below shows one of the Flash files working!

(VIDEO)

Now all I need to do it check that this will work on eight different computers at uni and also print out eight markers on paper, as markers on paper are supposed to work a lot better.

Performance Video Project – Finished Animations with Music

These are the animations that will be shown on each of the computers.

I combined the music and imagery in the same way for each of them but for some reason some of them have a pause in their loop whereas others are continuous, so I’m not really sure how to solve this, as I did exactly the same for each one and when the music and imagery were separate, all the loops worked fine with no gaps.

Performance Video Project – Music/Loops

I chose eight loops (two for each instrument) from Garageband that were in the same key and placed them all together so that the timing was all random to see if the loops would be able to work with each other in my piece, as they will be selected at random times by the markers and by different people. These eight loops seemed to work, so I exported each one as a separate mp3 and then used one with each of my animations. Below is the track with all the loops together:

Performance Video Project – 3D Mushrooms Complete!

Here are the 3D versions of my mushrooms:

Performance Video Project – Researching ARtoolkit

I did some research on ARtoolkit as FLARtoolkit was based on this, so I thought there may be a possibility that I may be able to use ARtoolkit to use multiple markers. I came across this:

http://en.wikipedia.org/wiki/ATOMIC_Authoring_Tool

which appears to be an interface that someone has created that enables you to do really simplistic augmented reality. However, to install it onto a mac it appears to be quite complex and you have to install something called Fink through terminal and as I don’t know what that is I don’t want to install it as doing things in terminal can possibly mess up the system files. Thus I watched the tutorial video for the software to see what it was capable of. Unfortunately, it seemed that it could do no better than FLARtoolkit and so I decided not to use it.

I then did some more research into the origins of ARtoolkit. It seems that ARtoolkit it just some sort of code library and to use it on a mac you need to use Xcode, which makes it far more cumbersome to use than FLARtoolkit as Xcode is essentially a compiler and so you have to even work out the code for showing the video in Xcode as it’s not automatically linked an any kind of video output like Flash is.

Performance Video Project – Development of Idea

After talking to both Liam and Phil and discovering that the coding in Flash for augmented reality that produces videos using multiple markers was far too difficult for any of us to work out easily, I decided that I would change my idea slightly to get around this problem. Thus, instead of trying to use multiple markers on one computer, I’m going to use a number of computers to display a video using one marker, so that way I don’t have to work out any new coding, I just have to load each computer with a different Flash project, each containing a different video. Also, I believe this may appear actually more of a ‘performance’ than my previous idea, because the audience will be essentially surrounded by the computers rather than just looking at a flat projection and this also makes it seem as if each computer/mushroom character is actually a member of a band.

Another change I decided to make was to cut the number of characters from five to four. This is because of two reasons. The first is because of the change written above, for I will need to use ten computers just to create a variety of loops that the audience can choose from as each character will have two loops and there are five characters so ten computers will be needed. The second reason is because, after looking at www.inbflat.net, I decided that the loops would be best made in a similar way to how it’s done on that site – so that it produces non uniform experimental music. Thus I decided that it was best to take out the drum loops and so I have merged the piano character and drum character into one, so that it looks like the drum mushroom but will represent piano loops. This is because drums or anything that makes a beat will make it difficult to stop and start at any time as it makes the music follow a rhythm and so if you hold up a drum marker at the wrong time, it may just sound like a mess.

Performance Video Project – Coding Problems and Mushroom Character Designs

I started looking at the Flash coding that would make augmented reality work. However, it turned out to be more complex than I imagined and I couldn’t find any appropriate tutorials online, so I decided I would make the mushroom characters first, so that the animations will be out of the way. The thing with the coding is, it’s easy to make a video appear, as you can just use the flARtoolkit coding, but it’s difficult to then add sound and make multiple things appear simultaneously.

I looked everywhere and for some reason no one has tried to make multiple VIDEOS appear! They have only used multiple markers with CUBES that are generated through papervision 3D (one of the tools needed to make augmented reality work in Flash), and so I can’t adjust the code to display VIDEOS because code that generates cubes is completely different.

Also, for some reason, when I tried playing my own video (using the default flARtoolkit settings) the sound in the video started playing before I even held up the marker! So that is another problem that I need to combat!

Thus, I made these mushroom designs:

This was my initial design idea. However, I decided that the eyes really didn’t suit the characters so I changed them:

They will be the ‘buttons’ of my instrument. They each represent a different sound, bass, synth/melody, guitar, piano and drums respectively. Also, I tried to have a sort of colour scheme that they all followed, so, as they’re mushrooms, I used yellows to reds to browns. I also coloured them in relation to their pitch, where the melody/synth has the highest pitch and therefore the lightest colour and the bass the lowest pitch and therefore the darkest colour.

Also, in case you were wondering how the fourth one is a mushroom, here is a photo of one in real life:

Performance Video Project – How Mushrooms Relate to Instruments

The images below should explain why I chose to make mushroom characters as the ‘buttons’ of the instrument I’m making.

The above is a typical flat mushroom.

And this is the ‘button’ of an instrument.

Can you now see why I chose mushrooms to represent the ‘buttons’ of my instrument?

Performance Video Project – Idea

Out of the all the different ways of interacting with video that I have seen lately, I felt I was most impressed by augmented reality. Augmented reality is about bringing video/animations into reality by the use of webcams and markers. It brings users much closer to the videos and characters because they can see themselves holding the characters/contents of the video and use a marker to make the video appear where ever they want it to appear.

My idea is to use augmented reality to combine visuals and music to make a live performance. It will be a group performance where each of the users will be responsible for an element of the music, each using a different instrument triggered by the augmented reality markers that they each hold up. The instruments will appear as little characters rather than the actual instruments themselves because I think this will make the whole performance a musical instrument in a sense, as each of the characters basically represent the buttons/frets/keys on an actual musical instrument. To make them seem more like part of an instrument, I have decided to make them follow a general theme where they all essentially look the same but will have a defining feature that reflects whichever instrument they represent. The instruments which they must represent are: bass (bassline), guitar, piano, synth and drums, for they are the instruments most commonly used in live performances (excluding orchestral). Thus, I decided to go with making their appearances look like mushrooms, as a mushroom is a very simplistic character design, so it will enable me to make lots of them, which is what I need to do. Each marker will trigger a different tune made by a different instrument. So essentially, the performance will be a group of people mixing sequences/beats and loops together live to form a musical track.

After doing a bit of research and with the help of Matt who did augmented reality in his previous group project, I found that one of the simplest ways of making augmented reality work was to download the script that many people use to make augmented reality work, then replace the video files with my own animations. However, the script does not have the coding to make animations appear simultaneously, so I will have to do some research into that and hopefully manage to get that to work, because my idea is quite reliant on this. Otherwise, I may have to use many different computers in order to make the characters appear simultaneously.

For those that are not sure about what augmented reality is, here is a good example video I found: