Between the Signal

Posted: October 21, 2026 | Author: Shihua Wang | Filed under: Uncategorized |Leave a comment »

In this project, I used webcam cameras and MediaPipe hand tracking technology to create an interactive TouchDesigner experience. The main limitation is that it’s not possible to use a keyboard or mouse; therefore, I decided to use the participants’ hands as the interface. Different hand movements trigger visual changes, and the participants gradually realize that their own bodies are actually controlling the entire environment, thereby revealing another layer of meaning associated with that environment.

Concept and Mystery

The core idea of this project is that the cameras are not merely used to display the participants themselves, but act like a damaged communication system that attempts to connect with another space. Initially, participants cannot fully understand the effects of their hand movements. As they continue to experiment with different gestures, the visual effects change, and those hidden digital worlds begin to become visible.

Therefore, this puzzle isn’t presented in the form of a traditional puzzle; instead, it allows participants to learn about this system through their own bodies. Their hands become the bridge between the physical world and the hidden digital world.

I chose to use hand tracking technology because I wanted this interaction to be more immersive. Clicking the mouse feels like controlling software, while moving hands in front of the camera makes the interaction more direct. Participants can immediately see the impact of their actions on the system, thereby making this process more personal.

Story and Audio

The atmosphere was inspired by Hideo Kojima’s Death Stranding 2: On the Beach. I was especially interested in its themes of connection, isolation, and uncertainty about whether connecting two different places is always a positive thing.

I used music from Death Stranding 2 because I wanted the sound to make the experience feel like the participant was discovering an unfamiliar space rather than simply testing a hand-tracking program. The slow and atmospheric music also helped connect the different visual transformations into one continuous experience.

Reflection

During the demonstration, people were particularly interested in how gestures control various changes. They also asked about the stories behind this experience. These feedbacks made me realize that the interaction itself is understandable, but the descriptions of the related stories can be conveyed more clearly.

If I continue to develop this project, I will add more demonstration elements, so that each gesture can reveal different aspects of the hidden world. Additionally, I will make the sounds more directly related to the participants’ movements, so that both visual and auditory systems can respond together to body movements.

---