The Integration of Traditional Culture and Modern Technology

For Pressure Project 2 — A Mystery Is Revealed, I created an interactive experience that combines traditional Chinese shadow puppetry with a story from Journey to the West. My inspiration came from the classic episode Sun Wukong （The Monkey King）and the White Bone Demon. By combining traditional Chinese storytelling and art with modern interactive technology, I wanted the audience to experience the fun of controlling shadow puppets themselves rather than simply watching a performance.

My project focuses on revealing the hidden identity of a mysterious woman. At the beginning of the experience, the audience first sees a landscape inspired by traditional Chinese art. A mysterious woman then appears, followed by Sun Wukong, the Monkey King. The audience can control Sun Wukong and try to uncover the woman’s secret. At the same time, another participant can control the mysterious woman, so the experience can be played by either one or two people. When Sun Wukong’s golden staff strikes the woman’s head, her human appearance gradually breaks apart, revealing the skeletal form hidden beneath her disguise. At that moment, the audience discovers that the seemingly ordinary woman is actually the White Bone Demon, and the mystery is finally revealed.

Process and Challenges

During the development process, the first problem I needed to solve was how to make the static shadow puppet characters move. To recreate the movable joints of traditional Chinese shadow puppets, I separated Sun Wukong’s character image into different parts, including his head, body, two arms, and two legs. I then imported these parts into TouchDesigner and used Transform TOPs to adjust their positions, sizes, and pivot points before putting them back together as a complete character. This step was more complicated than I originally expected. If the pivot points were not positioned correctly, the arms and legs would separate from the body when they rotated. I spent quite a lot of time making adjustments until the character’s movements looked more natural.

After assembling the character, I used MediaPipe hand tracking to create the interaction. Participants can control Sun Wukong with their left hand and the mysterious woman with their right hand. By bending different fingers, they can also control Sun Wukong’s head, arms, and legs. Both characters can move around the screen and automatically turn to face different directions based on hand movements. However, I encountered some unexpected challenges while developing these interactions. First, the hand-tracking system was very sensitive to small finger movements. Even when I tried to keep my fingers still, the character’s joints would continue shaking. Another problem was that MediaPipe did not always detect the two hands in the same order, which sometimes caused the characters controlled by the left and right hands to get mixed up. To solve these problems, I adjusted the hand recognition logic and used Lag CHOPs to smooth the characters’ movements, making the controls more stable.

Another major challenge was detecting collisions between Sun Wukong’s golden staff and the White Bone Demon. At first, I tried to detect attacks based on the distance between the characters and the rotation angle of Sun Wukong’s arm. However, this method was not very accurate. Sometimes, simply raising Sun Wukong’s arm would trigger the transformation, even when the staff had not actually touched the woman. To fix this, I redesigned the collision detection system by creating separate collision areas for the tip of the golden staff and the woman’s head. The transformation is now triggered only when these two areas overlap. When the staff successfully hits her head, her human appearance gradually distorts, revealing the skeletal form hidden underneath.

Through this project, I realized that interaction design is not just about making a computer recognize human movements. More importantly, the system needs to respond to those movements in a clear and meaningful way. I also learned that even small details can have a big impact on the user experience. Because of this, I had to keep testing and adjusting my project to reduce errors and make the interaction feel smoother.

Another thing I had to think about was how to make a simple character transformation feel like a moment when a mystery is truly revealed, rather than just another animation effect. To achieve this, I added opening subtitles, a question about the mysterious woman’s identity, and background music. I wanted these elements to create a sense of mystery and make the audience curious at the beginning, so they could eventually discover the truth through their own actions.

In traditional Chinese shadow puppetry, the audience usually watches performers control the puppets. However, in my project, the audience can become the puppeteers themselves. I think this change from watching to actively participating shows how modern interactive technology can bring a new kind of experience to traditional art forms. At the same time, I hope my project can encourage people who are unfamiliar with Journey to the West to become interested in Chinese traditional stories and culture, and perhaps learn more about them.

Pressure Project 2 – A Mystery is Revealed

My original intention in creating this project was to develop an interactive visual experience that allows users to control flames with their hands. I drew inspiration from K/DA’s new song “Firestarter,” particularly the song’s energetic vibe and its flame-related visual effects. I wanted to create a sensation of manipulating flames or magical energy—not through a keyboard or mouse, but by interacting directly with hand gestures.

I first used MediaPipe to track the position of my hands via the camera, then wrote Python code in a Script CHOP to recognize different gestures. For example, making a fist makes the flames disappear, and clapping changes the color of the flames. I also designed a new gesture: when one hand is raised and the other is lowered, the flame transforms into a fireball. I used the Sphere SOP and Switch SOP to change the shape of the flame emitter, and then used the Transform SOP to make the fireball follow the position between my hands. At first, the fireball’s movement was very erratic, so I later added a Lag CHOP to smooth out the tracking data.

The flame itself was created using NVIDIA Flow. I adjusted parameters such as speed, smoke, temperature, and fuel in the NV Flow Emitter to achieve the effect I was looking for. I also used the HSV Adjust TOP to change the flame’s color. Rather than a completely realistic flame, I wanted it to have a colorful, energetic visual effect, since my inspiration came from musical performances.

Another part of the project was creating a transformation effect. When the user holds both hands open for five seconds, a countdown starts, and the screen eventually changes into a thermal-camera-style view. I used Timer CHOP to control the timing and Python to make sure the interaction only triggers once. For the thermal effect, I combined Monochrome TOP, Lookup TOP, Ramp TOP, and Edge TOP, and used Switch TOP to change between the regular camera view and the thermal visualization. I also added music and fire sound effects using Audio File In CHOP and Audio Device Out CHOP.

One of the biggest challenges was getting all the gestures to work consistently. Sometimes MediaPipe would incorrectly detect a hand movement, which caused the timers to restart or the fireball to jump around. I spent quite a lot of time adjusting the Python code and the tracking settings to make everything work together.

Overall, I had a lot of fun experimenting with this project. I especially liked seeing how a simple hand movement could create such a dramatic visual change. This project helped me understand how different TouchDesigner operators can work together to create an interactive experience, and it made me want to explore more gesture-based interactions in the future.

Pressure Project 2 – New Age CD Player

My intention with this project was to create an interactive music experience. I was inspired by CD players, specifically the Fisher Slim 4000 (a CD player that allows the listener to see the disc spinning, while also having the capability to change between CDs) and basic CDJs (which allow for pitch correction of discs, sort of like a record player).

My node setup is fairly straightforward, and is driven by a set of switches. I have a switch for my audio and a switch for my images, both of which are controller by the same data (“nose X” in MediaPipe). This allows the user to cycle between the 5 songs I provided depending on where they are standing. Further, I incorporated pitch correction by using the nose Y in MediaPipe, using the value to change the speed of the audio files. To sync the audio speed the with speed in which each disc turns, I added a transform TOP after my image switch, with the rotation driven by a ramp LFO. I referenced nose Y to the LFO speed, which makes the CD turn faster or slower depending on the speed of the audio.

In terms of aesthetics, I designed all my visual elements in Adobe Photoshop, looking to make them as engaging as possible. I chose my songs first (I opted for more electronic sounds that would sound the most interesting when pitch corrected), and then designed each CD based on the vibe of each track. I originally wanted to create some feedback visualizers behind the discs, however, instead of complex loops I settled on a simple loop that increases in size. This resembled a CD player tray, which I liked, and also helped to emphasize the main disc. I framed the whole scene with some speaker designs, which helps to keep the attention on the center.

4/5 of my CD designs (the 5th one would not load)

During the demonstration, some of the movement cues were not apparent. I think this is because the projects we did in class focused mainly on hand tracking, while this experience involved body tracking. However, once everyone realized how the system works, everything ran smoothly.

Between the Signal

In this project, I used webcam cameras and MediaPipe hand tracking technology to create an interactive TouchDesigner experience. The main limitation is that it’s not possible to use a keyboard or mouse; therefore, I decided to use the participants’ hands as the interface. Different hand movements trigger visual changes, and the participants gradually realize that their own bodies are actually controlling the entire environment, thereby revealing another layer of meaning associated with that environment.

Concept and Mystery

The core idea of this project is that the cameras are not merely used to display the participants themselves, but act like a damaged communication system that attempts to connect with another space. Initially, participants cannot fully understand the effects of their hand movements. As they continue to experiment with different gestures, the visual effects change, and those hidden digital worlds begin to become visible.

Therefore, this puzzle isn’t presented in the form of a traditional puzzle; instead, it allows participants to learn about this system through their own bodies. Their hands become the bridge between the physical world and the hidden digital world.

I chose to use hand tracking technology because I wanted this interaction to be more immersive. Clicking the mouse feels like controlling software, while moving hands in front of the camera makes the interaction more direct. Participants can immediately see the impact of their actions on the system, thereby making this process more personal.

Story and Audio

The atmosphere was inspired by Hideo Kojima’s Death Stranding 2: On the Beach. I was especially interested in its themes of connection, isolation, and uncertainty about whether connecting two different places is always a positive thing.

I used music from Death Stranding 2 because I wanted the sound to make the experience feel like the participant was discovering an unfamiliar space rather than simply testing a hand-tracking program. The slow and atmospheric music also helped connect the different visual transformations into one continuous experience.

Reflection

During the demonstration, people were particularly interested in how gestures control various changes. They also asked about the stories behind this experience. These feedbacks made me realize that the interaction itself is understandable, but the descriptions of the related stories can be conveyed more clearly.

If I continue to develop this project, I will add more demonstration elements, so that each gesture can reveal different aspects of the hidden world. Additionally, I will make the sounds more directly related to the participants’ movements, so that both visual and auditory systems can respond together to body movements.

Pressure Project: Audio Visualizer

The goal of our first pressure project was to create a self-generating patch, given the time limit
of five hours. I was stuck at first, due to the openness of the project. But, I knew that when
starting in the class, I wanted to make an audio visualizer, so that is what I tried to do.

The time constraint didn’t feel like a massive issue, because I knew this pressure project was
more about exploring the different tools and seeing what we could make with a base
understanding. I had a very loose understanding of touch design, with only a few basic nodes
under my belt, so I knew I needed to start by watching some tutorials if I wanted to make a full
blown audio visualizer. I watched a few different videos, letting me a better understanding of
how certain audio and visual nodes work. I was able to mess with the values along the way, and
combine certain knowledge from each video to make something that was mine. I could choose
what I felt looked best and where in my composition, giving me so much room to work with.

What surprised me the most about this first pressure project was the sheer amount of different
nodes you can work with in touch designer. Given all of these nodes, and all of the different
combinations of the nodes, I can’t imagine how many different programs you could run with this
application. I felt like I grew a lot and got more understanding of the potential and opportunities
with touch designers, working mostly with the audio nodes and how they can affect the visual
nodes, but I am really looking forward to what I can do with the application in the future and how
far I can push my knowledge of it!

Pressure Project: The Self-Generating Patch

Description

The pressure project is a project that has a time constraint. This one has a 5 hour time constraint that you have to build a self-running visual system built in TouchDesigner. Once it starts, it runs on its own and needs no input. A noise source passes through a chain of edge detection, feedback loops, and displacement, then gets keyed and colorized. The result is a field of deep blue with a glowing magenta central radial, and a bright white area that rotates with the blue aspect. Three LFOs create a circle in the upper layer, so the magenta shape keeps pulsing and drifting and the feedback never settles into a repeat. You can watch it for a minute and not see the same frame twice. The only input is a keyboard shortcut by pressing 1 sends a pulse that resets the system.

Process Reflection

How did you approach the 5-hour time constraint?

I used the five hours to practice skills I wanted to grow instead of repeating what I already knew. We learned LFOs in class, so I layered them over a noise-based feedback system, which gave me a lot of motion from a small number of nodes. I stopped when time was nearly up and I felt the piece could hold attention. I had hoped to expand the LFO circles or arrange them into a cleaner pattern.

What choices did you make about what to include or exclude?

I included feedback loops, edge detection, and displacement because they keep simple noise changing indefinitely, which lets the system run alone. I left out audio, since getting two visual systems to work together took most of my hours. I also kept the system loose and abstract instead of patterned, which made it less predictable. The reset key used with the node keyboard in was my only interactive element.

What surprised you during the process?

How unpredictable the system is. Feedback responds strongly to small changes, so I could steer it but never fully control it. In class, people watched longer than I expected, even without audio. They seemed to enjoy not knowing what would happen next. The result differs from my original idea, but I’m happy with it. With more time, I’d clean up the visuals and add audio.

Which achievements did you unlock?

I earned the 30+ seconds of attention achievement. People in class stayed with the piece for over 30 seconds, which I didn’t expect, given there was no sound. I think the unpredictability was what kept them watching.

Pressure Project – Stimulation

For my self-generating patch, I was primarily focused on learning new techniques and getting familiar with Touch Designer as a software. I wanted my project to be colorful and stimulating, but I am also unfamiliar with this program, so I was limited with my own techniques. 

For the five hour time constraint, I spent most of my time doing trial and error with this project. For the first couple of hours, I was trying to figure out what to even do. I was consistently making and then deleting nodes to figure out what I wanted from this project. Eventually, I moved on to outsourcing other people’s projects and tutorials in order to actually learn more about what I was doing. Because we only learned a shallow amount of Touch Designer in class, I had to rely heavily on other tutorials in order to properly figure out how to navigate through the software. Once I found a few adequate videos to follow along with, I used that as a starting point for my project. I spent the remainder of my time limit following along with tutorials on easy visuals and then subsequently made them my own. I added color and shape changes so that it wasn’t just a copy and paste version of the tutorial. From there, I also added some of my own visuals from techniques that I already knew and had the hand of. Finally, I put everything together and made sure I had multiple visual stimuli so that it wasn’t just one shape or color that was changing. 

The main choice throughout this project was choosing what to add and how to do it. I wanted more than just one or two things, so I tried to add a multitude of different nodes and objects that would catch the viewers attention. I didn’t want to add too much, though, in order to not overstimulate the viewer. 

Overall, this project was surprisingly difficult and I wish I had more time to learn more about the software and how to make things in it. I am disappointed with the lack of information I had going into this project and I felt as though there wasn’t enough time given to properly execute a project that I would be proud of.