Cycle 2: The Rapture

For Cycle Two, I wanted to build off the kernel of my first cycle, Isadora in a theatrical context, and put my money where my mouth is by using it in a real production. My first project was exploratory; this one was functional. I wanted to program a show I could tour anywhere that had a screen. I had just revised The Phil Mitchell Radio Hour, a solo show booked for performances April 11th and 12th at UpFront Performance Space in Columbus, and at the Atlanta Fringe Festival May 27th–June 8th. Both venues had access to a projector, so I decided to build a visual component to the show using Isadora.

This meant I needed to create a projection design and keep it user friendly, since I wouldn’t be running the projections myself. It was important to me that the visual media didn’t just exist as a cool bonus, but felt like a performer alongside me, heightening the story, rather than distracting from it. I’d say the project was a major success. I want to tweak a few small moments before Atlanta, but the tweaks are minor and the list is short.

My Score:

Resources:

• Isadora, Adobe Premiere Pro, Adobe Express
  
• Projector, HDMI cable, laptop
  
• UpFront Performance Space
  
• Projector screen
  

Steps:

1. Create the “scenes” of the show in Isadora
   
2. Remaster sound design and integrate into scenes using enter triggers
   
3. Design projections
   
4. Add projections to scenes and build new scenes as needed
   
5. Run the full show and make sure it all holds together
   

For context: Here is the official show synopsis followed by an image from the original production:

“The Phil Mitchell Radio Hour follows a self-assured televangelist delivering a live sermon on divine pruning—only to realize mid-broadcast that the rapture is happening, and he didn’t make the cut. As bizarre callers flood the airwaves and chaos erupts, Phil scrambles to save face, his faith, and maybe even his soul—all while on live on air.”

I’m a big image guy, so I sort of skipped step one and jumped straight to step three. I began by designing what would become the “background” projection during moments of neutrality, places in the show where I wanted the projection to exist but not call too much attention to itself. I made a version of this in Adobe Express and pretty quickly realized that the stillness of the image made the show feel static. I exported a few versions and used Premiere to create an MP4 that would keep the energy alive while maintaining that “neutral” feeling. The result is a slow-moving gradient that shifts gently as the show goes on.

My light design for the original production was based around a red recording light and a blue tone meant to feel calming, those became my two dominant projection colors. Everything else built off that color story. I also created a recurring visual gag for the phrase “The Words of Triumph,” which was a highlight for audiences. 

Once the media was built, I moved into Isadora. This was where the real work of shaping began. While importing the MP4s, I also began remastering the sound design. Some of it had to be rebuilt entirely, and some was carried over from previous drafts as placeholders. The show had 58 cues. I won’t list them all here, but one challenge I faced was how to get the little red dot to appear throughout the show. I had initially imagined it as an MP4 that I could chroma key over a blank scene then recall with activate/deactivate scene actors.  Alex helped me realize that masking was the best solution, and once I got the hang of that, it opened up a new level of design.

All in all, I spent about two weeks on the project. The visual media and updated sound design took about 8–10 hours. Working in Isadora took another 3. By then, I felt really confident navigating the software. It was more about refining what I already understood. I initialized nearly every scene with specific values so it would be consistent regardless of who was operating. I also built some buttons for the stage manager in key moments. For example, when Phil yells, “Did I just see a horse?! Do horses go to Heaven?!” the stage manager pressed a button which activates a horse neighing sound.

When it came time to present a sample, I struggled with which moment to show. I chose a segment just before the first emergency alert of the Rapture, a phone call exchange. I thought it was the strongest standalone choice. It features three different projections (caller, emergency alert, and blank screen), has layered sound design, and doesn’t require much setup to understand.

Most of the feedback I received was about the performance itself, which surprised me. I was trying to showcase the Isadora work, but because I didn’t have a stage manager, I had to operate the cues myself, this created some awkward blocking. One suggestion I appreciated was to use a presenter’s remote in the future, so I could trigger cues without disrupting the flow of the scene. There were also some great questions about Phil’s awareness of the projections, should he see them? Should he acknowledge them? That question has stuck with me and influenced how I’ve shaped the rest of the show. Now, rather than pointing to the projections, Phil reacts as though they are happening in real time, letting the audience do the noticing. 

Overall, I’m extremely proud of this work. Audience members who had seen earlier drafts of the show commented on how much the projections and sound enhanced the piece. This clip represents the performance version of what was shown in class. Enjoy 🙂

Cycle 2: Up in Flames

               For this project, I wanted to keep stepping up the interactivity and visual polish. I knew I wouldn’t be able to get all the way with either, but my aim was to continue to step up those two aspects. In these cycles, my overall aim isn’t to create one, final project, but to use them as opportunities to continue to play and create as much as I can before the semester ends, and I have to return all the toys to the Lab. I didn’t do a great job of keeping track of how I was spending time on this project. I spent less time than I have in the past looking for assets, partially because I’m learning how to use my resources better and more quickly. I found the music I used more quickly than I expected, which also cut down on this aspect significantly. I spent most of my time increasing my understanding of actors I’d used before and trying to use them in new ways. Once I’d locked in the main visual, I figured out what to add to increase the interest and then invested some time in designing the iterations of the “embers.”

Exploding Embers

               I explored the Luminance Key, Calc Brightness, Inside Range, and Sequential Triggers actors, mostly, seeing what I could figure out with them. I used them in the last cycle, but I wanted to see if I could use them in a different way that would require less precision from the depth sensor, which was also at play in this project. In the last cycle, I struggled with the depth sensor not being robust enough to tune into the very exact way I wanted to use it; this time, I figured I would use it more to capture motion than precision. I still have to use TouchDesigner to get the sensor to work with Isadora. Because it’s free, I can see myself continuing to explore TouchDesigner after this class to see if I can at least replicate the kinds of things I’ve been exploring in these cycles. For the purposes of class, though, I’m trying to fit as much Isadora in as I can before I have to give the fob back!

Actor Setup

               I think the main challenge continues to be that I want the sensor to be more robust than it is. I think, in many ways, I’m running into hardware issues all around. My computer struggles to keep up with the things I’m creating. But I am working with my resources to create meaningful experiences as best I can.

               With this project, I wanted to add enough to it that it wasn’t immediately discernable exactly how everything was working. In that, I feel I succeeded. While it was eventually all discovered, it did take a few rounds of the song that was playing, and, even then, there were some uncertainties. Sound continues to be a bit of an afterthought for me and is something I want to more consciously tie into my last cycle, if I can leave myself enough time to do so. The thing I didn’t include that I had an impulse toward was to connect the sound to the sensor, once I actually added it. This was an option that was pointed out during the presentation, so it seems that that was definitely an opportunity for enhancement. I also think there was a desire to be able to control more of the experience in some way, to be able to affect it more. I think this would have been somewhat mitigated if it was, indeed, tied to the music in some way.

               The presentation itself did immediately inspire movement and interaction, which was my main hope for it. While I did set up that getting close to it would be better, I also think that the class felt driven to do that somewhat on their own. I think choice of color, music, and movement helped a lot with that. The class was able to pick up on the fire aspect of it, as well as the fact that some part of it was reacting to their movement. They also were able to pick up on the fact that the dots were changing size beyond the pulsing they do when in stasis. I think the one aspect that wasn’t brought up or mentioned was that the intensity of the dots was also tied into the sensor. It was brighter when left alone to draw people to it and got less intense when things were closer to the sensor. A kind of inverse relationship. In some ways, the reverse would have encouraged interaction, but I was also interested in how people might be drawn to come interact with this if it weren’t in a presentation and landed on this solution.

               If I were to take this further, I definitely like the idea of tying the embers to different pieces of the sound/song. There was also an interest in increasing the scale of the presentation, which would also be something to explore. I think my resources limit my ability to do the sound aspect, at least, but it would be the step I would take if I were going to go further with this, somehow. I also liked the idea of different scenes, and it is definitely something I was thinking about and would have been interested in adding with more time. I can absolutely see this as a thirty-minute, rotating scene interactive thing. I think the final note was that the boxes around the exploding embers could be smoothed out. It’s an issue I’ve run into with other projects and something I would definitely like to address in future projects.

Video of Stage

Song Used for Presentation

Pressure Project 3: Going to the Movies

For pressure project 3, I set out to tell a clear story using only sound. I’d done a similar project once before a handful of years ago and was in the middle of teaching sound design to my Introduction to Production Design class, so I felt fortunate to have a little experience and to be in the correct mindset. I wanted to make sure there was a clear narrative flow and that the narrative was fleshed out completely with details. I opted for a realistic soundscape, a journey, because it felt, to me, the most logical way to organize the sounds in the way I wanted to.

In terms of what it would be, I started broadly thinking about what culture thing I might create a story for. As I mentioned in class, I did originally consider doing a kind of ancestral origin story. I first considered doing a story about/for my great grandparents who came to the United States from Croatia. The initial thought was to follow them from a small town there, across the ocean, to Ellis Island, where they landed, and then perhaps onto the train and to Milwaukee (even though that skips a portion of their story). Ultimately, part of why I abandoned this idea was because it felt too sweeping for two minutes, and I started to realize that a lot of what I would want for that story would have to be recorded, rather than found.

In the end, I decided to think much smaller, about a cultural experience about me, specifically. Because I still had this idea of a “journey” in my head, I came to the idea of walking to the movies. Being able to walk to an independent movie house, preferably an old one, is something I look for when we’re moving somewhere new, so it felt like the right way to go. I decided to age up the movie theatre in my head a bit to play up the soundscape; quiet chairs and doors didn’t seem as interesting as someplace that needs a healthy round of WD40. The theatres for the last three places I’ve lived are pictured below.

Neighborhood Theatres

With a story in mind, I set out to figure out how to actually make the sound story. With a decent amount of digging around, I finally remembered that the program I’d used in the past was Audacity, so I at least knew it would be free and relatively accessible. From there, I searched out sounds on the sound effects websites I’d given my students to use (YouTube Sound Library, Freesound, Free Music Archive, etc.). I used Audacity to get the sound file from the movie I wanted to use at the end of the piece, which from The Women (1939). The main challenge I faced was that I’ve personally only worked with sound once before maybe four years ago when I took a class very much like the one I’m teaching now. The other is that it’s always harder than you want it to be to find the right sound you’re looking for. I was mindful of the relatively limited time I had to work on the project and tried to work as quickly as I could to make a list of sound I felt I needed and try to track them down. I also challenged myself to make the sound locational in the piece, to work with levels and placement so that it felt as if it was actually happening to the listener. In some ways, this was the easiest part of the project to do. For me, it was easy to pick out when a sound wasn’t where I wanted it to be or something was too loud or quiet to my mind’s eye.

I didn’t keep careful track of how my time worked out or was split up. I did the project, more or less, straight through, moving between tasks as it made sense. I designed it mostly from start to finish, rather than jumping around, finding a sound, placing it, and editing it. I shifted lengths and things like that as I went, and I spent maybe half an hour at the end fine tuning things and making sure the timing made sense.

A Smattering of the Sounds

During the presentation, the class understood that a journey was happening and, eventually, that we were at the movies. They experienced this realization at slightly different points, though it universally seemed to happen once we had gone through the theatre doors and into the theatre itself. They reported that the levels and sound placement worked, that they knew they were inside of an enclosed space. I think the variety of information was also pleasing, in terms of the inclusion of social interactions, environmental information, the transitions between places, etc. I think the inclusion of a visual would have made the sound story come into focus much more quickly. If I had included a picture of a movie theatre, or, better, the painting I had in my head (below), I think the whole journey would have been extremely clear. As it was, once I explained it, I think all the choices made sense and told the story I was trying to tell.

Overall, I think the project was successful, though perhaps in need of visual accompaniment, particularly if this kind of movie going isn’t your usual thing. I also think one of the effects, the popcorn machine in the lobby, needed adjusting; I was worried about it being too loud and overcorrected.

New York Movie (Edward Hopper, 1939)

Pressure Project 3: Surfing Soundwaves

When starting this project, it was a bit daunting and overwhelming to decide which part of my cultural heritage to explore. Theatre initially came to mind, but that felt too easy, I wanted to choose something that would allow me to share a new side of myself with my peers. With graduation on the horizon, I’ve been thinking a lot about home. I grew up near the water, and there’s this unbreakable need in me to be close to it. That’s where I began: sourcing sounds of the ocean on Pixabay. I went so specific as to only use sounds from the Atlantic Ocean when possible, even finding one clip from Florida.. But I realized that the ocean itself didn’t fully capture my heritage, it was a setting, not a story. I had to go deeper.

I began thinking about my dad, who was a professional surfer. I’m sure he had aspirations for me to follow a similar path. I felt a lot of pressure to surf as a kid, it was honestly terrifying. But over time, and on my own terms, I grew to love it. Now it’s how my dad and I connect. We sit out on the water together, talk, and when the waves come, we each ride one in, knowing we’ll meet back up. The more I thought about that ritual, the more homesick I felt. Even though the assignment was meant to be sound-based, I found myself needing a visual anchor. I dug through my camera roll and found a photo that captured the calmness and rhythm of beach life, something soft and personal that helped ground my process.

Once I had that anchor, I started refining the design itself. Since this was primarily a sonic piece, I spent about thirty minutes watching first-person surfing videos. It was a surprisingly helpful strategy for identifying key sounds I usually overlook. From that research, I pinpointed four essential moments: walking on sand, stepping into water, submerging, and the subtle fade of the beach as you move away from it. While wave sounds and paddling were expected, those smaller, transitional sounds became crucial to conveying movement and environment.

I then spent another chunk of time combing through Pixabay and ended up selecting sixteen different sounds to track the journey from the car to the ocean. But as I started laying them out, it became clear that I wasn’t telling a story yet, I was just building an immersive environment.

Avoiding that realization for a while, I focused instead on the technical side. I mapped out a soundscape where the waves panned from right to left, putting the listener in the middle of the ocean. I also created a longform panning effect for the beach crowd, which built up as you walk from the parking lot but slowly faded once you were in the water. I was proud of how the spatial relationships evolved, but I knew something was still missing.

Late in the process, I returned to the idea of my dad. I created an opening sequence with a sound of submerging followed by a child gasping for air. I used a kid’s breath to imply inexperience, maybe fear. To mark a shift in time, I recorded a voicemail similar to the kinds of messages my dad and I send each other, with the general sentiment“I’ll see you in the water.” It was meant to signal growth, like the child in the beginning was now the adult. I had hoped to use my actual dad’s voicemail, but when I called him, I found his box wasn’t set up.

I decided to end the piece with the paddling slowing down and the waves settling as I/the character reached dad. I repeated “dad” a couple of times—not in panic, just to get his attention, to make it clear he was present and alive. I used very little dialogue overall, letting the sound do most of the storytelling, but I tried to be economical with the few words I did include.

During the presentation, I felt confident in the immersive quality of the work, but I wasn’t prepared for how the tone might be read. The feedback I received was incredibly insightful. While the environment came through clearly, the emotional tone felt ambiguous. Several people thought the piece might be about someone who had passed away, either the child or the father. I had added a soft, ethereal track underneath the ocean to evoke memory, but that layer created a blurred, melancholic vibe. One person brought up the odd urgency in the paddling sounds, which I completely agreed with. I had tried to make my own paddling sounds in my kitchen sink, but they didn’t sound believable, so I settled on a track from Pixabay. 

Looking back, I know I missed the mark slightly with this piece. It’s hard to convey the feeling of home when you’re far from it, and I got caught up in the sonic texture at the expense of a clearer narrative. That said, I still stand by the story. It’s true to me. When I shared it with my wife, she immediately recognized the voicemail moment and said, “I’ve heard those exact messages before, so that makes sense.” 

If I had more time, I would revisit the paddling and find a better way to include my dad’s voice, either through an actual message or a recorded conversation. That addition alone might have clarified the tone and ensured that people knew this was a story about connection, not loss.

Cycle 1: Big Rainbow Piano

For Cycle 1, I started by identifying a few resources I wanted to explore. Since attending the Touch Designer workshop at the beginning of the semester, I have been searching for an opportunity to utilize the software and explore the tools that were provided to us from Sudo Magic. I was able to get my hands on the “Geo Zone Detector” Comp which allows you to easily draw a rectangular volume and use the point cloud data from a depth sensor to detect when a person or object is occupying that zone. 

I decided to use the basic framework of a human sized piano keyboard to test this system. I began by drawing the “geo zones” in the Comp to be approximately the size of a person, one meter by two meters by three meters. This was roughly the width and height of the person with enough length to occupy a human sized “key”. Then the next step was to align the depth sensor in the Motion Lab. As part of the “patch lab” in the Touch Designer workshop, I learned to use the point transform Top to orient the pint cloud to the Motion Lab x,y,z coordinate system. 

Once I had the point cloud and spatial trigger boxes aligned and working in unison, I tried to attach a specific sound frequency to each “geo zone”. I started by using the audio oscillator chop. This produced a pure sine wave that I could dial in the exact frequency of each note on the piano. Getting this to work with the Geo Zone Comp was straight forward, but I did have to alter the settings for zone “trigger threshold” so that it would detect only an arm of a leg and not just the whole person.  

The pure sine wave was able to prove that the concept was working. However, I was disappointed in the tonality of the sound and I wanted to find a way to trigger an actual piano. I wanted to find a solution that wasn’t too complicated and did not require too much additional hardware. I have dabbled with Garage Band for many years and I was well aware of the plethora of virtual instruments available in the software. My previous experience with a MIDI keyboard lead me believe this would be a plug and play type of solution. 

Unfortunately, the world of MIDI and Apple drivers is not exactly “plug and play” but there is a pretty easy to use “audio midi setup” app that allows you to see the devices plugged into your computer. A few google searches led me to the IAC or inter-application communication driver. This allows one software program to send MIDI messages to another without the need for MIDI hardware attached. This is exactly what I needed. However, the MIDI out Chop in Touch Designer does not fill in the necessary coding to talk directly to other devices. A few additional google searches led me to the simple channel plus note syntax. Once I inserted all of the triggers in this format, the Geo Zone Detector was triggering a MIDI keyboard in Garage Band. Success! I spent over an hour listening to different synths and virtual instruments before I landed on “Gritty Bells” which had a pleasant tone, but also a rhythmic component that essentially “played a song” when you hit different notes together.

In an effort to connect the Isadora skills I had already learned, I wanted to draw some graphical boxes similar to the “shapes” actor. I found the “rectangle” sop to be essentially the same thing. You can draw a box and also define a line color, fill color and line weight. This process created the “keys” of the keyboard. In addition to triggering MIDI notes, I connected the same triggers to the ”fill” parameter of the boxes. The end result was the key lighting up and playing the notes simultaneously.  

Finally, I projected the boxes on the ground and tweaked the keystone to line up with the boxes already defined in the Geo Zone Detector. I decided to roll out a strip of white Marley to help the top down projection reflect off of the floor and then refined the keystone and geo zones further so they aligned. This was a bit of a trial and error operation. The Marley was a fixed size so  I started by aligning the projections and then scaled the Geo Zones to match. I could easily tweak each section, or the entire point cloud using the point transform Top. 

When it came time to present, I added some color projection on the MOLA cylinder scrim. These scrims were already in place for Noah’s cycle 1, So I decided at the last minute to add a rainbow gradient projection onto the cylinder. I had this idea previously and I got lucky having just enough prep time to quickly make the gradient match the keyboard and load it into Resolume utilizing a previous projection map (resource).

I made everyone leave the lab before they saw the finished cycle 1. This helped to reset the vibe after the somewhat chaotic setup process. Some people had already seen the basic setup before but there were also some new guests that have never experienced the installation or the motion lab. Everyone seemed to timidly approach the installation but once the were all there, everyone took turns playing notes and testing the reactivity of the “piano”. Some people tried to play a song and others were trying to find the limits of the depth sensor to see how little of their presence triggered a note and how high or low their foot or hand needed to be.  

The feedback I received was generally positive. People enjoyed the colorful and playful nature of the setup as well as the pleasant tones emitted my the speakers. Someone said it reminded them of a Fischer Price Xylophone, which I must admit, was not something I was going for, but unintentionally recreated almost exactly! Some of the other feedback I received was that they enjoyed that they could step back and watch others interact and play since there were essentially a “play” space on the Marley, and a “observer” space outside the rectangle. Some others commented that more interactive possibilities would have been interesting, such as a different response to the height of your hand or foot that triggered the notes.  

For Cycle 2 I plan on using the same basic concept but abstract the entire field to be a little less rectangular and obviously a type of piano. I have been experimenting with a spatial synthesizer that can utilize the immersive sound system in the Motion Lab. I also plan to add more “geo zone” boxes to increase the sensitivity and interactive possibilities. 

Cycle 1: A Touch of Magic

            For this project, my primary strategy was to try to reverse engineer a piece of media I experienced in a queue at Disney World. I was pretty sure I understood how it was working, but I was curious if I would be able to reconstruct some portion of it on my own. After thinking about the patterns, I knew the media sequence was a series of scenes that were just variations on one another with either some kind of count trigger or time marker that triggered the movement into the next scene. I also wanted to use this as an opportunity to explore a depth sensor, as a tactic to experiment with other interactive media I’ve seen used in various capacities. The two together were challenging, and I think there were ways that I could have created a more polished finished product if I had focused on one thing or another. I think this is a byproduct of me originally imagining a much bigger/flashier end product at the end of the three cycles, which I have since scaled back from significantly. In the end, these cycles will end up serving more as vehicles for me to continue to learn and explore so I can better recognize how these tools and technologies are working out in the wild, thereby making me a better partner to the people who are actually executing the media side of things.

Figure 1: No Time for Making Things Neat

The tools I used for this were a depth sensor, Touch Designer mainly because Isadora couldn’t recognize the depth sensor without it, and Isadora. In Isadora, I used a Virtual Stage for the first time in one of my projects. It helped me be able to see where the areas were I was isolating with the depth sensor, while keeping it separate from the area where the interactive pieces were being triggered. I used the Luminance Key, NDI Watcher, and Calc Brightness actors for the first time in a project and continued to enjoy the possibilities of triggers, gates, and sequential triggers. I find that I am always thinking about how the story progresses, what happens next, and in building in ways for what I’m creating to naturally bring up another scene or next thing once it’s reached a relatively low threshold or interactions or time.

Figure 2: An Abundance of Gates and Triggers

Some challenges were that I didn’t quite author to the technology, which caused some issues, and I didn’t fully understand the obstacles I might encounter in moving from one space to another. The very controlled environment of a ride queue is perfect for these kinds of tools, and I feel like I understand much better, now, how this could work in a more controlled and defined space. I also think focusing on the challenges of a depth sensor alone and maybe using it to more carefully capture motion, rather than necessarily a range of depth, could have allowed this to be more sensitive and, therefore, successful. I certainly spent more time than I would have liked trying to figure out how to get the trigger to be less sensitive than I wanted it to be but sensitive enough to still fire, and I still didn’t quite figure it out.

I think the decision to trigger both a visual and auditory reaction when the specific areas were interacted with was a good one. The sound, especially, seemed like a success with my classmates interacted with it, and the simplistic nature of a little bit of magic on a black screen also worked well. It was noted that the specific visual and sound combination did make those playing with it want to do more magical motions or incited joy, so I think that those were good choices.

Figure 3: A Touch of Magic

I can’t help wishing the presentation would have been visually more polished, but I don’t really have the skills to make more finalized video product. Adobe stock leaves something to be desired, but it’s what I’m mostly working with right now, as well as whatever sounds I can find in a relatively short amount of time that kind of “do the job.” I think this is where my scenic design abilities need the other, media design part to fully execute the ideas that are in my head. As I said, though, I am primarily using these cycles to continue to explore and learn how things work.

            In the end, I think the project elicited the responses I was hoping for, which mostly was for people to realize a motion in a specific space would trigger magic. I could also see how some magic incited the desire to seek out more, and I think that a more comprehensive version of this could absolutely be extrapolated out into the queue entertainment I saw at Disney.* I do think this project was successful, in that I learned what I set out to learn and it incited play, and, depending on how the coming cycles develop and what I learn, there feels like the possibility some of this might show up again in the coming cycles.

Figure 4: The To Be Continued Scene Transition

*In the queue, guests interacted with shadows of either butterflies, bells, or Tinkerbell. After a certain amount of time or a certain number of interactions, the scene would kind of “explode” into either a flurry of butterflies, the bells ringing madly, or Tinkerbell fluttering all around, and move into the next scene. Guests’ shadows would be visible in some scenes and instances and not in others, sometimes a hat would appear on a shadow’s head. I think I mostly figured out what was going on in the queue sequence, and for that, I’m very proud of myself.