Pressure Project 2 – Webcam Theremin

For Pressure Project 2, I decided to make a theremin out of a webcam!

For the visual component, I was inspired by one of Chuck Csuri’s works, his Structural Fragmentation 2021.

The Isadora patch I created uses the total brightness of the webcam feed to decide whether to play sound, as well as how high/low the pitch should be. The webcam is placed on a table looking up at the ceiling, and a person is encouraged to use their hand to interact with it. The idea was that the video feed from the webcam would be detectably darker when a hand was placed over the lens, and the closer the hand was to the lens, the darker it would be. The fact that the vast majority of webcams have automatic brightness correction complicated this idea, but it still functioned mostly as intended.

As for the sonic element, I recorded myself singing an “oo” vowel so as to continue with the theme from Chuck Csuri’s piece. This sound was pitched up and down according to the distance the “performer” had their hand from the camera. Unfortunately, I think the vowel ended up not sounding identifiably like an “oo” vowel, or like a human voice at all, and instead ended up just sounding like a synthesized tone.

Visually, the Isadora patch starts creating Os on the screen when a hand is detected by the webcam. The Os are created on the left side and slide to the right side as they dissipate over time, leaving a clear trail. The higher the pitch of the theremin, the higher the Os are created on the screen (vertically), and vice versa. The color of the Os is also shifted according to the pitch of the theremin.

Cycle 2: Stove Maps

For my cycle 2, I was mostly focused on how I was physically going to present my outputs in motion lab. Since I had already created a main mechanism (my stove), I wanted to start work on the more visual element of the project.

In addition to the stove, I had decided to add a sink element, as this was another interactive piece of a kitchen/house I felt I could incorporate. I used Shadertoy to find a simple water GLSL shader that I connected to my sink user actor. When activated by pressing the tinfoil of the “faucet,” the GLSL shader would fade in, making the sink look full.

I used an alpha mask to achieve this effect. At this point, all of these “activations,” (like this and the stove burners) were happening by jump++ actors jumping to new scenes. This was a quick and dirty way of activating these elements, but I would soon have to update these to work more seamlessly within the same scene in order to simplify my patch and output.

Once I started working in motion lab, I first had to decide how I would handle the different output streams I had in my Isadora patch. I knew I needed at least two projectors: one that would show the background of the kitchen, and one that would show the stove top and sink.

Above are my sink and stove blocks–I had to be aware of how I could position them so that the top-down projector would still hit them. Now that they had been positioned, it was onto output stream management and mapping. I was using the Mac Pro in motion lab, and it took me a long time to sort out how to manage my outputs using the triple head and matrix switcher so that the correct images would project from the correct places. Not only did the matrix switcher have to be set correctly, but I had to fix my Isadora output so that each projection element was on its own stage. I got a lot of help from Alex on this, which was definitely needed.

Finally, I got it set so I could map. Above is my mapped sink. Even though it was mapping a square onto another square, it took a fair amount of time to find where exactly on the floor of motion lab the small sink was, and how I had to move it in order for it to be placed correctly.

From my presentation, I was finally able to present the idea of the space to the class. I got some valuable feedback in terms of switching the direction my stove and sink faced and decided to implement that going into cycle 3.

Cycle 3 (Amy)

Cycle 3: Ready for interaction – Title: ‘Undoing’

Artist Statement:

The roots of this project stem from a very personal experience. In October, unexpectedly, I was faced with the break down of an 8 year relationship. This caused a shattering of self which spun things out of control and messily permeated all areas of my life. It was a hard reset. I stopped eating. The ground disappeared beneath my feet and I could not summon any semblance of rationality. It also erased the need to cling to complex processes and technique which I had previously used to hide behind. It demanded a stripping down to raw honesty in myself and my work that felt both unfamiliar and deeply uncomfortable. I did not want to forget.

Grounded in Pauline Oliveros’s philosophy on deep listening, I explore the dissolving of self as an experience in the form of an interactive audio-visual installation. I believe productivity stems from the human desire for proof of existence which causes us to do, undo, and redo over and over again. To challenge this notion, I invite viewers to let go of the need to do and simply ‘be’ as they are relieved of their burden of self in the tangle of others. Simultaneously overwhelming and gentle, the outcome of the work is indeterminate and depends on the willingness of the viewer to engage and shape their own experience.

As a demonstration of the interdependence and power of interconnectedness, this project exists both also as a large-scale collective experience for 10+ participants, where the actions of each person impacts the overall outcome, as well as a solitary experience for one participant alone with headphones. Further development in Cycle 4 and beyond will include exploration into different spaces both inside and outside, framing the piece in ritual by using tactile materials such as sand, and ornamentation on lightbulbs as well as other sources of light reactivity. As a performance piece, I would like to explore methods of generative improvisation through the use of DAWs, as well as a purely acoustic version where people sing in a room together. 

Reflection:

In this iteration, I was mainly interested in the responses of the participant. I devised several versions of this piece for various groups with the intention of creating a different experience for each situation. As the maker, instead of my usual eagerness for validation, I took a step back and observed. I was curious to see how people interacted with the piece out of context. Did they approach with apprehension? Were they able to be playful? How long did they stay for? Did they grasp for control? Did they allow for a collaborative experience? Were they able to take it in?

The more people engaged, the more I realized that the feedback of each participant told me was more about what kind of person they were than the actual piece. This was such a gift! I learned so much about what it means to have an experience and how to cultivate a sense of freedom and agency in a work.

In terms of the piece itself, for Cycle 3 I mainly worked on refining the layering of voices and how they would fit into the shape of a mandala (for the rug!). For the final presentation in the motion lab, the sound formed a 5 layer mandala. I asked a vocalist friend (a true bass!) to record some very low tones for me. I placed these tones in the very center of the mandala and then gradually expanded outward from high vocal intensity to gentle sustaining tones to breathing on the outmost circle. During our final DEMS critique, I was thrilled to see how spontaneous and playful everyone was in their approach to the piece. Pure JOY!

Untangle – loosen – untie – undo is the mantra I have adopted through the creation of this work. In contrast to my previous need to finish a work and then determine it good or bad, I hope to continue developing iterations of this project embraced by the idea that we too exist in cycles.

Cycle 2 (Amy)

Cycle 2: Implementation and expansion – working title: ‘Annihilation of self’

November 14th goal: Create a piece that allows for a fluid collective experience for the class. 

Inspiration: ‘Deep listening takes us below the surface of our consciousness and helps to change or dissolve limiting boundaries.’ – Pauline Oliveros

Implementation: acquisition of resources and programming

• Light –
• 8 dimmable lightbulbs
• 8 socket bases
• Chauvet DJ DMX 4-4 channel (lightbulbs > Enttec)
• Enttec USB Pro (DMX > Computer)
• Motion –
• Orbbec Depth Sensor (installed on ceiling of Motion Lab)
• Sound –
• Solo voice improvisation, recording, layering
• 2 Portable Speakers

Reflection:

Developing from Cycle 1, mostly a sketch using motion detection, translating to sound and light reactivity, Cycle 2 really fleshed out the the piece and set up for a collective experience in preparation for Cycle 3.

A crucial addition was CG’s fantastic 11ft circular rug! It provided the perfect layout for our group and served as a physical ‘score’ for the piece. To detect motion, we hung an Orbbec from the ceiling facing the floor, which provided a bird’s-eye-view for my reactivity map. As people stepped onto the rug, the Orbbec would trigger sound and light in response to their position and movements. In Isadora, the mechanisms for operation were: OpenNI (Orbbec detection) > Luminance key > Alpha Mask (for circular crop) > Eyes++ > blob detection, which then controlled the volume of sound files on loop and levels of brightness for the DMX output.

The biggest challenge for this iteration was the set up. Because each lightbulb required its own extension cord, the set up and tear down became quite labor intensive and each took about an hour. It was definitely worth the effort. The lightbulbs were entrancing! Seeing the initial concept of light and sound ‘waking up’ to physical presence was utterly magical. Much credit goes to Nico for generously providing mesmerizing dance as a quick demo. Thanks Nico!

Cycle 1 (Amy)

Cycle 1: research and experiments, Initial title – ‘the things you don’t see’

October 11th goal: I would like for someone’s presence, just being, to be sensed. They are reacting to the system reacting to them, it’s reciprocated, ongoing. The feedback loop of interaction.

Research: keywords and quotes from Quantum Listening by Pauline Oliveros

• ‘Listening field…’ – pg 13
• ‘Listening to their [audience] listening.’ – pg 16
• ‘Deep listening takes us below the surface of our consciousness and helps to change or dissolve limiting boundaries.’ – pg 38
• ‘What if you could hear the frequency of colors?’ – pg 40

Reflection:

In this initial stage, my desires were a jumble. I had the vague idea of creating an interactive audio-visual installation that ‘woke up’ to physical presence and responded generatively to subtle bodily movements. I was also very intrigued by the concept of deep listening and the philosophy of Pauline Oliveros during this time, particularly, her work ‘Teach yourself to Fly.’ In this work, a group of people sing together, responding to and diverging from the voices of those around them. The result is an organic sound cloud that is ever shifting, highlighting breath, fluidity, and a state of impermanence.

Alex provided guidance by helping me sift through my thoughts and in constructing a clear directive. The first was – how do I use the Orbbec to pick up movement and how is that reflected through programming in Isadora? Second – how do I transfer this response onto a lightbulb using a DMX box? Third – how does the motion translate or manipulate sound output?

In Cycle 1, I programmed the bare bones of motion and sound reactivity in Isadora. AND got the lightbulbs to work, yay!

Cycle 3: Xylophone Hero at Fort Izzy

This Cycle I decided to focus on the physical presentation of my project. My first task was to connect my xylophone to Isadora using the Makey Makey. I wrapped foil around each key of the xylophone, then made “wires” out of foil that connected to each foiled key. Then I remapped the back inputs of the Makey Makey to the numbers 1-8 and attached the breadboard wires to the Makey Makey. I attached the Makey Makey to the inside of a shoebox and fed the breadboard wires through to the outside of the box. From there, I connected the alligator clips and the breadboard wires. To ground the mallet, I tied one end of some conductive string to Earth on the Makey Makey and tied the other end around the handle of the mallet, taping the extra string to the mallet. I then covered the top of the mallet in foil.

Before my Cycle 3 presentation, I mapped the output of my Isadora patch onto the xylophone and I set up a “blanket fort” around my projection in the Motion Lab. I also went through my patch to make sure everything was working as expected, and I added a start screen, an end screen, and a restart. I staged the fort with stuffed animals, playing cards, and other items that might be found in a children’s bedroom.

I am extremely happy with how this project turned out and I am so proud of what I produced. If I were to do more cycles, I would paint the shoebox to look more like a toy box, secure the bottom sheet to the floor, and I would figure out a way to hold a high score or display a scoreboard. I could also figure out a way to introduce game difficulties.

Cycle 2: Keeping Score

In between Cycle 1 and Cycle 2, I decided that my input device would be a children’s xylophone, which I would make work using the Makey Makey.

During Cycle 2, I spent most of my time working in Isadora. I wanted to make sure I had my Isadora patch working the way I wanted before I started any of the physical presentation elements. To start, I turned my four arrow shapes into rectangles, then added four more to simulate a xylophone.

Next, I added in a way for the computer to tell the player which key to hit. To do this I started by duplicating the user actors I had made for the previous rectangles. Each user actor is almost identical to the blinking arrows in Cycle 1, but has input from a Get Global Values Actor inside the User Actor, and an Inside Range Actor outside of the User Actor. The outside mechanism uses an Enter Scene Trigger, Timed Trigger, Trigger Value, Pulse Generator, Activate and Deactivate Scene Actors, the Random Actor and the Float to Integer Actor, as well as an Inside Range Actor connected to each of the duplicated User Actors. Both patch snippets are pictured below.

I added in a goal for the game; To score as many points as possible in one minute. First I added a way for Isadora to track the player’s score using the Counter. Actor. Then I added in a game timer using the TimerTool Actor downloaded from the Add-Ons page of the TroikaTronics website.

I also tried to add in a scoreboard feature, but it turned out to be much more difficult than I had anticipated, so a scoreboard is likely a Cycle 4 type of addition.

In Cycle 3 I am going to spend more time focusing on the physical presentation of my project. My goals are to get the xylophone connected to my Isadora patch, figure out how I want to set up in the Motion Lab, and projection map my Stage onto the xylophone.

Cycle 3

After I studied connecting Isadora to Arduino and experimented its potentials in Cycle 1 and 2, I focused on considering what type of experience I could create with that technology in Cycle 3.
Feedbacks from everyone in Cycle 2, that just a tiny motion of small servo motors and its fuzzy structure could have its own quality, really inspired me.

I decided not to make such a “well designed” machine but just collected found objects and combined them with mechanics.
It was like installing an electric life in dead objects; how static objects get its another life as an interactive/kinetic machine.

Reactions from viewers (not only in our class but I also showed it to Art grad students and faculties) exceeded my expectations; the interactive motion which responds to viewers but still keeps a certain unpredictability made the sense of life (this object is living its own life) in each of viewers.
While its rule of interaction is very very very simple (just chasing and responding to a viewer in front of a webcam), viewers grows the sense and the meaning of that object in their own way…it was my discovery through this Cycle 3.
It could be a powerful potential of such an interactive project a creator doesn’t need fill out all the concept and meaning but viewers would find and grow it through the relationship with that object.

Body as Reservoir of Secrets, DEMS Cycle 3

In cycle 3, the three things I focused on were:

1. Refined the composition of the graphics generated. Included more gaussian blur effects to reduce the number of hard-edges present in the work. The reduction of hard edges was intentional to reduce the “digital” nature of the graphics.
   
2. I continued to develop the costume. I drew, painted and stitched in the costume, to create a more shamanistic leaning for the costume, hoping to steer away from an immediate aviation or space travel reference.
   
3. I managed to program a Hot Folder in my computer, so that syncs up with the stage to image actor on isadora, and whatever Isadora exports to the Capture folder is automatically printed.

Decisions made:

1. I decided to include an additional web-cam to do a live-capture of the pile of printed matter that was generated. The pile would then appear in some print outs.
   
2. I moved the pulse sensing contact mic to my wrist due to the headache I experienced after the cycle 2 presentation.
   
3. The score of the piece was supposed to be live-painting/collaging. There was not enough space/time for that to happen at ACCAD and the dec 8th critique. So I devised a score that was about folding each sheet of paper 8 times, and being unable to fold it the 9th time as a parallel to the irretrievable nature of ancestry.

Challenges/Feedback:

1. My patch got very complicated and towards the end I was having some difficulty figuring how everything was wired, even though I did make an effort to be organized.
   
2. Isadora crashed from time to time. I will look into buying a high capacity windows laptop next year.
   
3. I needed some troubleshooting with the Hot Folder and took awhile to figure out why I was generating so many prints. Turned out that I had an earlier version isadora file running in the background which was feeding the hot folder a bunch of images.
   
4. The prints generated had a general purple hue to them. It seems like I needed to vary the colors more in the graphics generated, so that this purple hue is counteracted/balanced.

Observations/ Ongoing curiosities:

1. The printer working generated an unexpected mechanical soundscape to the piece. The rhythmic and hasty machine sound creates a sense of urgent-ness to the work, I find myself feeling some anxiousness and felt pressure to work at that rhythm/pace. I felt “chased” by the printer.
   
2. A classmate posed me a question about generative art, and how generative art tend not to have any material cost to it. She asked me how I felt about the paper and ink expended for this performance. I actually welcome the creation of these materials because as an artist working in collage, I hardly throw away anything because I save everything in flat files. These materials will eventually find its way into another piece of work in the future.
   
3. I intend to stage a cycle 4 in the Sherman Studio Arts Center after the semester quietens down. Cycle 4 will have the live-painting aspect and I will be exploring using what I have created in this class, as means to create a video work. I am also curious about how feature extraction can influence sound and be used for composing.