Video Visual Jockey

The goal of pressure project three was to create a patch, or collection of patches, that are controlled through a MIDI device. The controls should be used to create a visual experience through music, like a DJ but for visuals, a visual jockey. 

We were given a MIDI control board to test out, and we mainly focused on using the knobs and sliders on the board to transform and manipulate 3D models, and a brief test on lighting. I felt I had a good knowledge of those nodes and wanted to expand my knowledge with other nodes. 

I enjoy video editing, and I knew TouchDesigner had built-in filter options, so I decided to use the MIDI controller to manipulate filters, speed, and effects on the videos I input. 

My knowledge was slim to start out, so I lightly followed a tutorial on using the pixilate controller node. This ended up teaching me about other nodes like matte, edge, and a few other nodes that can be used to manipulate others. 

I used this base knowledge of how filters work to toy with other filters like time machine and change to color. I feel like as much as tutorials are helpful for specific issues, toying around with TouchDesigner is such a major part of furthering your knowledge and creating something that is new and unique.

I wanted mine to feel unique to me, so I used different videos that I found resonated with me or my interests, while also keeping the emotion in mind of the songs it would be used for. I knew matching the visuals to the feeling of the audio was a major part, so I wanted to make sure I covered all bases when it came to matching the vibe of the audio. I used the switch node, activated with blending, to glide through my videos.

I toyed with the idea of incorporating 3D visuals overtop or between the videos, but I found I just couldn’t really get the moving visuals of my filters to cohesively flow with any sort of particles or full models. I had tried to overlay the video visuals on top of models, but it just felt too messy and not really visually appealing. 

My final project resulted in the blending of the filters and videos made through the use of the MIDI keyboard. The colors, visuals, speed, and other effects can match the vibe of the song depending on how good the visual jockey is, which I definitley have more progress to make. Next time, I want to try and map my board more clearly and focus on how it can manipulate my visuals more. Overall, I just want to be a better visual jockey.

Link to Pressure Project #3

A Game of Cat and Mouse

The goal of this pressure project was to create a way for human and computer to communicate, that isn’t a way that is typical like keyboards and mice, but rather using body movements to cause a reaction.

When working through this pressure project, I started out wanting to make a program where you were able to use certain facial and hand tracking to activate certain noises. The goal was to be able to make a “beat” with different movements of my fingers and my face. Little did I know, this would be me setting my goals too high. I got the initial sounds to work, but it just did not sound or feel like an enjoyable experience. This led me back to the drawing board.

My second attempt was more successful. I ended up making a game for me to play with my cat. The controls were pretty simple, using the webcam to track the movements of my two pointer fingers and whether my mouth was open or closed. I used the values generated by the node tracking my fingers and mouth to activate other nodes. These nodes included an audio node that activates a quack when you open your mouth and plays a soft mouse chattering when your mouth is closed. The other major nodes that I used were the transform and 3D model nodes, allowing me to use the values of where my fingers were in relation to the bounds of the camera to the bounds of the 3D model of a mouse on the program’s canvas.

When using my projector to cast my program on a surface, I found that the more contrast, the easier it was for my cat to see. So I ended up going with a silhouette look for the mouse, allowing my cat to see it more clearly. In the end, I felt like I learned a lot about the difficulties and possibilities of body tracking, and how it can be used to create unique and engaging experiences for others, even if they are not human.

Link to Pressure Project #2 Download

From Ink to Aurora: A MIDI-Controlled Video Synth

Project Description

My Video Synth is a real-time audiovisual instrument built in TouchDesigner and operated with a Korg nanoKONTROL2 MIDI controller. It combines three visual modules: Ink, Stars, and Aurora.

Ink produces monochrome radial lines, Stars creates a field of particles and connections, and Aurora introduces colorful glowing forms and distortion. The visuals respond to a shared music input, while I control their visibility, composition, and intensity during the performance. I designed and built three visual modules in TouchDesigner: Ink, Stars, and Aurora. Ink explores monochrome radial lines, Stars uses particles and connections to create a sense of space, and Aurora introduces colorful glowing forms and distortion. I combined these modules into one instrument with a shared audio input and eight MIDI controls, allowing me to layer and transform the visuals in real time.

The setup consists of a computer running TouchDesigner, a Korg nanoKONTROL2 connected through USB.Each visual module sits inside its own component. Its output passes through transformation and brightness controls before entering a shared composite. A black background and a master brightness stage complete the final output.

A single Audio File In CHOP supplies music to all three modules through native CHOP connections. Each module retains its own audio-analysis network. This combines automatic musical responses with manual performance decisions.

The eight faders control:

The MIDI controls use native MIDI In, Select, and Math CHOPs. Math CHOPs convert the controller’s 0–127 values into suitable ranges for each visual parameter.

Music: https://youtu.be/YBhL2a7CKYQ?si=88yAjIZQOqE4Ia6v

Process Reflection

How did I design my modular instrument?

I separated the visuals into three independent modules so I could work on one effect without rebuilding the others. A common output structure made it possible to mix different visual styles within one instrument.I first tested the controls with on-screen sliders, then replaced their control signals with MIDI input. This allowed me to develop the visual system before connecting the hardware.

What did I choose to include or exclude?

I selected three visual systems from my own experiments because they offer contrasting textures: sharp lines, delicate particles, and soft luminous forms. I kept each system independent so I could perform with one at a time or combine them. I limited the interface to eight controls, prioritizing brightness, rotation, scale, position, and distortion. This kept the instrument manageable during a live performance while still allowing a wide range of visual combinations.

What surprised me when I performed?

I was surprised by how difficult it was to make visual decisions while keeping up with the music. Sometimes the rhythm changed faster than I could react. When performing to unfamiliar or randomly selected music, I wasn’t always sure which control to use or what visual change to make next. This made me realize how much live performance depends on practice and familiarity with both the music and the instrument.
During development, I discovered how strongly parameter ranges affect playability. Sending raw MIDI values directly to brightness or position could overexpose an image or move it off-screen. Correct scaling was essential to making the instrument controllable.

What would I change next time?

I would add smoother transitions between controller values, a unified restart control for the music and visual systems, and a clearer performance interface. I would also develop more original visual modules and explore controls that affect their internal behavior.Further rehearsal would help me refine the pacing: deciding when to introduce a new layer, when to reduce complexity, and when to leave a visual moment unchanged.

AFTERIMAGE-A Fully Functional MIDI-Controlled DJ Deck & Audiovisual Performance System

Overview

For this project, I created a fully functional digital DJ deck combined with a real-time visual performance system using TouchDesigner and a KORG nanoKONTROL2 MIDI controller. My goal was to create an interactive audiovisual experience where I could mix music, manipulate visual effects, and control the entire performance through a single physical controller.

I selected three songs by aespa: LEMONADE, Whiplash, and Armageddon, and connected them into a continuous DJ set. Instead of simply playing music alongside prerecorded visuals, I designed a system that allows me to control both audio and visuals in real time.

Inspiration

My inspiration came from aespa’s concerts. I have always been interested in how their performances combine electronic music, futuristic visual design, lighting, and digital effects to create a strong audiovisual atmosphere. I especially enjoy how the visuals change with the music, making the entire performance feel more immersive.

I wanted to recreate this sense of energy through my own interactive system, rather than copying the original concert visuals. Since I am interested in both music performance and interactive design, I decided to combine the functions of a DJ deck with a generative visual system, allowing me to perform the music and visuals at the same time.

I named the project AFTERIMAGE because the feedback and particle effects create visual trails that remain briefly on the screen, forming constantly changing layers of movement.

System Design

I divided my system into two main parts: audio mixing and visual control, both operated through the KORG nanoKONTROL2.

For the audio system, I used Audio File In CHOPs to load the three songs and Math CHOPs to create separate audio channels. I built a crossfading system controlled by MIDI sliders, allowing me to smoothly transition between tracks by adjusting their volume levels. I also incorporated playback triggers, bass adjustment, and a DJ-style siren effect to make the mixing process more flexible.

For the visual system, I created a generative particle environment with adjustable turbulence, force, noise, and feedback. I mapped MIDI knobs, sliders, and buttons to different parameters, allowing me to manipulate the direction and intensity of particle movement, as well as the persistence of visual trails. I also connected audio analysis to the particle system so that the visuals could react dynamically to the music.

One of my main design decisions was to make the audio and visual systems work together without making everything automatic. While the music influences the particle behavior, I can still manually change the visuals and control the transitions. This allows me to improvise during the performance and create different results each time.

For the visual system, I can control particle turbulence, upward and horizontal movement, and particle contraction into a single vortex. I also added controls for changing particle size during contraction and release, adjusting the size of disappearing particles, and switching particle colors. For the audio system, I mapped the controller to initialize the first track, LEMONADE, and transition between LEMONADE, Whiplash, and Armageddon. I also incorporated bass volume adjustment, a DJ scratching effect, and a siren sound effect. By combining these functions within one MIDI controller, I can manipulate particle behavior, create visual transitions, and perform live audio mixing without switching between different interfaces.

Reflection

One of the biggest challenges I faced during this project was integrating audio mixing and visual control into a single MIDI controller. Since I wanted to build a functional DJ deck rather than a simple music visualizer, I had to spend a lot of time experimenting with audio transitions, MIDI mappings, and the relationships between different parameters in TouchDesigner. I also realized that creating interesting visual effects was only part of the process. Making those effects easy to control during a live performance was equally important. Sometimes, even a small change in the particle system could produce an unexpected result, which made the process both frustrating and exciting. Through this project, I learned more about how to balance technical functionality with creative expression. If I continue developing AFTERIMAGE, I would like to improve the audio mixing features, add more visual effects, and make the transitions between different visual states smoother. Overall, this experience helped me understand how TouchDesigner can be used not only to generate visuals, but also to create an interactive instrument that combines my interests in music, performance, and digital design.

Neon Mirage

For this project, I created an interactive video synthesizer in TouchDesigner called Neon Mirage. My inspiration came from kaleidoscope toys and neon lights. I wanted to create a digital kaleidoscope with patterns, colors, and movements that could constantly change. This was my first time using a MIDI controller to control visuals in real time. I found it very interesting and also learned some new skills through this experience.

Technical Setup and Visual Design

I mainly used GLSL shaders and TOP operators in TouchDesigner to create the visuals. The main kaleidoscope is made of symmetrical patterns and flowing neon lines. I also created a separate layer of geometric fragments to make the patterns look more like moving pieces of glass. Another important part of my project is the particle system. The particles move in a spiral and spread outward, creating a glowing effect. I used feedback to give them light trails and make their movements feel smoother.

For the interaction, I used a Korg nanoKONTROL2 MIDI controller. Different faders control rotation speed, liquid distortion, fragment movement, fragment density, particle effects, and colors.

The color control includes five different palettes: pink and cyan, yellow and orange, green and blue, silver and white, and black and gold. I wanted the colors to change gradually instead of suddenly switching between presets.

Design Process and Challenges

I think that besides making the controls fun to use, finding a balance between visual complexity and clarity was also an important part of this project.

At first, I mainly focused on adding more patterns, colors, and effects because I thought they would make the kaleidoscope more interesting. However, when I combined too many effects, the screen became very crowded. The neon lines and glass fragments sometimes overlapped too much, making it difficult to see their individual shapes. I realized that adding more visual elements does not always make the final work better.

I also had problems with blurry particles and changing the original square composition into a widescreen format. During the process, I spent a lot of time adjusting shaders, testing MIDI controls, and comparing different visual effects. I tried to keep the details I liked while improving the parts that might distract the audience.

Performance and Reflection

Unlike creating a finished animation, using a video synthesizer requires me to constantly make adjustments based on the music while the visuals are running. I found this process very interesting. I needed to think about when to change the colors, when to add more particles, and when to keep the visuals simple. I also found that controlling several faders at the same time was not easy.

Designing a visual instrument is different from creating a beautiful image. I needed to consider not only how the visuals looked, but also whether I could easily control these effects during a live performance.

Reassemble Yourself: Revealing a Portrait Through Gesture

Project Description

Reassemble Yourself is an interactive portrait that invites participants to explore how their movements can transform their own image. The mystery is a question: How can two hands change the way we see ourselves? Instead of revealing a hidden object, the project reveals a relationship between the participant’s body and a fragmented digital portrait.

The experience begins with a full camera image. On-screen instructions ask participants to face the camera, then introduce their hands one at a time. This opening gives the interaction a clear starting point and explains the controls before the visual transformation begins.

How the Project Works

The project is built in TouchDesigner and uses MediaPipe to detect facial and hand landmarks from a live webcam feed. MediaPipe provides the positions of points such as the wrist, thumb tip, and index fingertip. A Python script turns these positions into control values by calculating the distance and angle between the thumb and index finger.

The two sides of the screen have different roles. The hand on the left controls the size and tilt of the image tiles. Opening or closing the fingers changes their size, while turning the hand changes their tilt. The hand on the right controls the tiles’ shape and rotation. Together, these controls allow participants to reshape their portrait through movement.

A GLSL shader processes the live camera image into a grid of geometric tiles. Each tile contains part of the moving image, so participants can still recognize themselves as the portrait changes. Circular details around the edges emphasize facial features and create additional points of attention.

The opening sequence waits for face detection, followed by the left-side hand and then the right-side hand. After both hands remain visible for approximately two seconds, the full image gradually transitions into the fragmented portrait. Gentle background music accompanies the experience and supports its exploratory atmosphere.

Reflection

One important design decision was to make the interaction softer. Early changes felt too fast, so I reduced the sensitivity and smoothed the control values. This allows the image to respond gradually and makes small movements easier to explore.

Adding the opening sequence also changed the structure of the experience. Previously, participants entered directly into the effect without a clear introduction. The new sequence establishes a beginning and introduces each hand’s role.

For me, the mystery is revealed through learning the relationship between gesture and image. Participants discover that the portrait is something they can continuously reshape. The project turns into their own face an opportunity for curiosity and playful experimentation.

DJ.toe — Playing the Flock

ShihuaWang XMD

Pressure Project 3: VJ Battle

DJ.toe is a real-time video synthesizer built in TouchDesigner and performed using a Korg nanoKONTROL2. It develops my earlier bird project into an instrument: a flock circles a central red bird, but its speed, density, scale of movement, and visual intensity can now change through physical controls. The patch does not play music locally. Instead, I listen to the music in the room and make the visual decisions myself.

Process Reflection

My starting point was a flock that already knew how to fly. The challenge was making it something I could play. In the earlier version, audio analysis controlled movement automatically. For this project, I disconnected that dependency and gave the birds a continuous flight speed before introducing MIDI. This mattered because I wanted the flock to keep moving even when no audio was entering my computer.

I organized the controller around actions I could remember without reading the node network. The eight faders control flight speed, gathering and dispersal, movement irregularity, bird count, fading, grayscale color, camera distance, and camera orbit speed. I separated dispersal from bird size because spreading a flock across the screen creates a different feeling from enlarging its individual bodies. Separating these decisions made the instrument easier to understand.

Initially, I wanted to avoid buttons because the setup already felt complicated. Later, I realized that a few buttons could provide accents that faders could not. I added a temporary size pulse and developed a held color effect: pressing the button introduces rapidly changing colors and brightness variation, while releasing it returns to the original image. A knob adjusts the flashing frequency. I treated these as interruptions rather than the permanent appearance of the piece.

The black-and-white image remained important to me. It gives the flock a clear silhouette and makes changes in density easy to see. It also gives color somewhere to arrive from. If everything is constantly colorful and intense, an additional effect has less impact. I wanted room for quieter moments, empty space, and gradual changes before a sudden visual accent.

The technical process involved several corrections. At one point, MIDI values were changing correctly, but the birds were not flying because the rotation export needed to be reconnected. Another attempt produced distortion without the color effect I wanted. I rebuilt that section around an explicit color mapping instead. These moments taught me to check each stage separately: controller input, numerical range, animation, and final display.

During the live performance, I was much more nervous than I had expected. Once the music started, though, I focused on listening and working the controller. I hit each drum beat with the visual changes I intended, and the controls worked smoothly throughout. The result felt even better than I had hoped. What surprised me most was how different the project felt when I was performing it: the birds were no longer just an animation I had built, but something I could play. Getting those moments right made the time spent adjusting and troubleshooting the controls feel worthwhile.

The most interesting shift was from watching a system to making decisions within it. The controller gave me a way to shape tension through accumulation, speed, distance, and interruption. For a future version, I would add clearer physical labels, a few reliable scene presets, and smoother transitions between them. I would also spend more time rehearsing when to leave the image alone. Building more effects is tempting, but learning when a small movement is enough is part of learning to perform this instrument.

Tools and Credits

• Software: TouchDesigner.
• Controller: Korg nanoKONTROL2.
• Bird assets: Model from my personal asset library, with rigging and flight poses prepared by me.

Groove Machine – Pressure Project 3

For my visual synth, I wanted to create two compositions that represented different vibes. My intention was to have the visuals begin slowly, transforming into a more intense visual scene as the song goes on. I was inspired by rave visuals, which have a very chaotic, colorful nature. 

The first patch is the simplest of the two, and is made up of a SOP feedback loop. First, I created a simple SOP composition using a copy node. I then ran this through a feedback loop that changed various transform parameters and some lighting adjustments. This allowed for quick feedback loop adjustment to create hypnotic, circular patterns. 

The second patch is a little more complex, and uses external images along with 3D shapes. The SOP composition is pretty similar to the first patch, but it utilizes 3D shapes, while also giving the ability for the user to switch between a circle and box shape. I wanted to make this patch more colorful, so I brought in some Photoshop designs I had done in the past and hooked them up to a switch, which cycles through the images depending on the speed of the LFO that controls the switch. These designs served as the background for the patch, but I also used them as textures for my SOP composition, which added more color and visual interest. 

After I finished my two main patches, my primary goal was focused on being able to switch between the two in an efficient manner. I was not aware that when adding a Geo, Light, and Camera, that every SOP in view gets rendered. This was not ideal when separating my two patches, so I hooked up my SOPs and final outputs to a switch, which allowed for MIDI control of both switches simultaneously. Further, there was some nice interplay between the two patches due to the feedback loops I used, which helped to blend together the final output. 

One thing that helped me with the live performance aspect was the use of LFOs rather than direct MIDI controls. During the in class demo, I realized that controlling every parameter through MIDI was not ideal as I was only able to turn a certain amount of knobs and sliders at a time. To make dynamic visuals that always had movement, I controlled most of my parameters with LFOs to ensure constant movement. For my MIDI, I mapped my controls to the speed of the LFOs rather than the parameters themselves, which allowed for easier, more efficient control. An example where this was most effective was through isolating parts of the song. Rather than continuously pulsing a parameter to emulate the bass or melody, I was able to beatmatch the parameter by changing the LFO, which let me focus on changing other parameters. 

You can see my full patch and screenshots using the link