From Ink to Aurora: A MIDI-Controlled Video Synth

Project Description

My Video Synth is a real-time audiovisual instrument built in TouchDesigner and operated with a Korg nanoKONTROL2 MIDI controller. It combines three visual modules: Ink, Stars, and Aurora.

Ink produces monochrome radial lines, Stars creates a field of particles and connections, and Aurora introduces colorful glowing forms and distortion. The visuals respond to a shared music input, while I control their visibility, composition, and intensity during the performance. I designed and built three visual modules in TouchDesigner: Ink, Stars, and Aurora. Ink explores monochrome radial lines, Stars uses particles and connections to create a sense of space, and Aurora introduces colorful glowing forms and distortion. I combined these modules into one instrument with a shared audio input and eight MIDI controls, allowing me to layer and transform the visuals in real time.

The setup consists of a computer running TouchDesigner, a Korg nanoKONTROL2 connected through USB.Each visual module sits inside its own component. Its output passes through transformation and brightness controls before entering a shared composite. A black background and a master brightness stage complete the final output.

A single Audio File In CHOP supplies music to all three modules through native CHOP connections. Each module retains its own audio-analysis network. This combines automatic musical responses with manual performance decisions.

The eight faders control:

The MIDI controls use native MIDI In, Select, and Math CHOPs. Math CHOPs convert the controller’s 0–127 values into suitable ranges for each visual parameter.

Music: https://youtu.be/YBhL2a7CKYQ?si=88yAjIZQOqE4Ia6v

Process Reflection

How did I design my modular instrument?

I separated the visuals into three independent modules so I could work on one effect without rebuilding the others. A common output structure made it possible to mix different visual styles within one instrument.I first tested the controls with on-screen sliders, then replaced their control signals with MIDI input. This allowed me to develop the visual system before connecting the hardware.

What did I choose to include or exclude?

I selected three visual systems from my own experiments because they offer contrasting textures: sharp lines, delicate particles, and soft luminous forms. I kept each system independent so I could perform with one at a time or combine them. I limited the interface to eight controls, prioritizing brightness, rotation, scale, position, and distortion. This kept the instrument manageable during a live performance while still allowing a wide range of visual combinations.

What surprised me when I performed?

I was surprised by how difficult it was to make visual decisions while keeping up with the music. Sometimes the rhythm changed faster than I could react. When performing to unfamiliar or randomly selected music, I wasn’t always sure which control to use or what visual change to make next. This made me realize how much live performance depends on practice and familiarity with both the music and the instrument.
During development, I discovered how strongly parameter ranges affect playability. Sending raw MIDI values directly to brightness or position could overexpose an image or move it off-screen. Correct scaling was essential to making the instrument controllable.

What would I change next time?

I would add smoother transitions between controller values, a unified restart control for the music and visual systems, and a clearer performance interface. I would also develop more original visual modules and explore controls that affect their internal behavior.Further rehearsal would help me refine the pacing: deciding when to introduce a new layer, when to reduce complexity, and when to leave a visual moment unchanged.

AFTERIMAGE-A Fully Functional MIDI-Controlled DJ Deck & Audiovisual Performance System

Overview

For this project, I created a fully functional digital DJ deck combined with a real-time visual performance system using TouchDesigner and a KORG nanoKONTROL2 MIDI controller. My goal was to create an interactive audiovisual experience where I could mix music, manipulate visual effects, and control the entire performance through a single physical controller.

I selected three songs by aespa: LEMONADE, Whiplash, and Armageddon, and connected them into a continuous DJ set. Instead of simply playing music alongside prerecorded visuals, I designed a system that allows me to control both audio and visuals in real time.

Inspiration

My inspiration came from aespa’s concerts. I have always been interested in how their performances combine electronic music, futuristic visual design, lighting, and digital effects to create a strong audiovisual atmosphere. I especially enjoy how the visuals change with the music, making the entire performance feel more immersive.

I wanted to recreate this sense of energy through my own interactive system, rather than copying the original concert visuals. Since I am interested in both music performance and interactive design, I decided to combine the functions of a DJ deck with a generative visual system, allowing me to perform the music and visuals at the same time.

I named the project AFTERIMAGE because the feedback and particle effects create visual trails that remain briefly on the screen, forming constantly changing layers of movement.

System Design

I divided my system into two main parts: audio mixing and visual control, both operated through the KORG nanoKONTROL2.

For the audio system, I used Audio File In CHOPs to load the three songs and Math CHOPs to create separate audio channels. I built a crossfading system controlled by MIDI sliders, allowing me to smoothly transition between tracks by adjusting their volume levels. I also incorporated playback triggers, bass adjustment, and a DJ-style siren effect to make the mixing process more flexible.

For the visual system, I created a generative particle environment with adjustable turbulence, force, noise, and feedback. I mapped MIDI knobs, sliders, and buttons to different parameters, allowing me to manipulate the direction and intensity of particle movement, as well as the persistence of visual trails. I also connected audio analysis to the particle system so that the visuals could react dynamically to the music.

One of my main design decisions was to make the audio and visual systems work together without making everything automatic. While the music influences the particle behavior, I can still manually change the visuals and control the transitions. This allows me to improvise during the performance and create different results each time.

For the visual system, I can control particle turbulence, upward and horizontal movement, and particle contraction into a single vortex. I also added controls for changing particle size during contraction and release, adjusting the size of disappearing particles, and switching particle colors. For the audio system, I mapped the controller to initialize the first track, LEMONADE, and transition between LEMONADE, Whiplash, and Armageddon. I also incorporated bass volume adjustment, a DJ scratching effect, and a siren sound effect. By combining these functions within one MIDI controller, I can manipulate particle behavior, create visual transitions, and perform live audio mixing without switching between different interfaces.

Reflection

One of the biggest challenges I faced during this project was integrating audio mixing and visual control into a single MIDI controller. Since I wanted to build a functional DJ deck rather than a simple music visualizer, I had to spend a lot of time experimenting with audio transitions, MIDI mappings, and the relationships between different parameters in TouchDesigner. I also realized that creating interesting visual effects was only part of the process. Making those effects easy to control during a live performance was equally important. Sometimes, even a small change in the particle system could produce an unexpected result, which made the process both frustrating and exciting. Through this project, I learned more about how to balance technical functionality with creative expression. If I continue developing AFTERIMAGE, I would like to improve the audio mixing features, add more visual effects, and make the transitions between different visual states smoother. Overall, this experience helped me understand how TouchDesigner can be used not only to generate visuals, but also to create an interactive instrument that combines my interests in music, performance, and digital design.

Neon Mirage

For this project, I created an interactive video synthesizer in TouchDesigner called Neon Mirage. My inspiration came from kaleidoscope toys and neon lights. I wanted to create a digital kaleidoscope with patterns, colors, and movements that could constantly change. This was my first time using a MIDI controller to control visuals in real time. I found it very interesting and also learned some new skills through this experience.

Technical Setup and Visual Design

I mainly used GLSL shaders and TOP operators in TouchDesigner to create the visuals. The main kaleidoscope is made of symmetrical patterns and flowing neon lines. I also created a separate layer of geometric fragments to make the patterns look more like moving pieces of glass. Another important part of my project is the particle system. The particles move in a spiral and spread outward, creating a glowing effect. I used feedback to give them light trails and make their movements feel smoother.

For the interaction, I used a Korg nanoKONTROL2 MIDI controller. Different faders control rotation speed, liquid distortion, fragment movement, fragment density, particle effects, and colors.

The color control includes five different palettes: pink and cyan, yellow and orange, green and blue, silver and white, and black and gold. I wanted the colors to change gradually instead of suddenly switching between presets.

Design Process and Challenges

I think that besides making the controls fun to use, finding a balance between visual complexity and clarity was also an important part of this project.

At first, I mainly focused on adding more patterns, colors, and effects because I thought they would make the kaleidoscope more interesting. However, when I combined too many effects, the screen became very crowded. The neon lines and glass fragments sometimes overlapped too much, making it difficult to see their individual shapes. I realized that adding more visual elements does not always make the final work better.

I also had problems with blurry particles and changing the original square composition into a widescreen format. During the process, I spent a lot of time adjusting shaders, testing MIDI controls, and comparing different visual effects. I tried to keep the details I liked while improving the parts that might distract the audience.

Performance and Reflection

Unlike creating a finished animation, using a video synthesizer requires me to constantly make adjustments based on the music while the visuals are running. I found this process very interesting. I needed to think about when to change the colors, when to add more particles, and when to keep the visuals simple. I also found that controlling several faders at the same time was not easy.

Designing a visual instrument is different from creating a beautiful image. I needed to consider not only how the visuals looked, but also whether I could easily control these effects during a live performance.

Reassemble Yourself: Revealing a Portrait Through Gesture

Project Description

Reassemble Yourself is an interactive portrait that invites participants to explore how their movements can transform their own image. The mystery is a question: How can two hands change the way we see ourselves? Instead of revealing a hidden object, the project reveals a relationship between the participant’s body and a fragmented digital portrait.

The experience begins with a full camera image. On-screen instructions ask participants to face the camera, then introduce their hands one at a time. This opening gives the interaction a clear starting point and explains the controls before the visual transformation begins.

How the Project Works

The project is built in TouchDesigner and uses MediaPipe to detect facial and hand landmarks from a live webcam feed. MediaPipe provides the positions of points such as the wrist, thumb tip, and index fingertip. A Python script turns these positions into control values by calculating the distance and angle between the thumb and index finger.

The two sides of the screen have different roles. The hand on the left controls the size and tilt of the image tiles. Opening or closing the fingers changes their size, while turning the hand changes their tilt. The hand on the right controls the tiles’ shape and rotation. Together, these controls allow participants to reshape their portrait through movement.

A GLSL shader processes the live camera image into a grid of geometric tiles. Each tile contains part of the moving image, so participants can still recognize themselves as the portrait changes. Circular details around the edges emphasize facial features and create additional points of attention.

The opening sequence waits for face detection, followed by the left-side hand and then the right-side hand. After both hands remain visible for approximately two seconds, the full image gradually transitions into the fragmented portrait. Gentle background music accompanies the experience and supports its exploratory atmosphere.

Reflection

One important design decision was to make the interaction softer. Early changes felt too fast, so I reduced the sensitivity and smoothed the control values. This allows the image to respond gradually and makes small movements easier to explore.

Adding the opening sequence also changed the structure of the experience. Previously, participants entered directly into the effect without a clear introduction. The new sequence establishes a beginning and introduces each hand’s role.

For me, the mystery is revealed through learning the relationship between gesture and image. Participants discover that the portrait is something they can continuously reshape. The project turns into their own face an opportunity for curiosity and playful experimentation.

DJ.toe — Playing the Flock

ShihuaWang XMD

Pressure Project 3: VJ Battle

DJ.toe is a real-time video synthesizer built in TouchDesigner and performed using a Korg nanoKONTROL2. It develops my earlier bird project into an instrument: a flock circles a central red bird, but its speed, density, scale of movement, and visual intensity can now change through physical controls. The patch does not play music locally. Instead, I listen to the music in the room and make the visual decisions myself.

Process Reflection

My starting point was a flock that already knew how to fly. The challenge was making it something I could play. In the earlier version, audio analysis controlled movement automatically. For this project, I disconnected that dependency and gave the birds a continuous flight speed before introducing MIDI. This mattered because I wanted the flock to keep moving even when no audio was entering my computer.

I organized the controller around actions I could remember without reading the node network. The eight faders control flight speed, gathering and dispersal, movement irregularity, bird count, fading, grayscale color, camera distance, and camera orbit speed. I separated dispersal from bird size because spreading a flock across the screen creates a different feeling from enlarging its individual bodies. Separating these decisions made the instrument easier to understand.

Initially, I wanted to avoid buttons because the setup already felt complicated. Later, I realized that a few buttons could provide accents that faders could not. I added a temporary size pulse and developed a held color effect: pressing the button introduces rapidly changing colors and brightness variation, while releasing it returns to the original image. A knob adjusts the flashing frequency. I treated these as interruptions rather than the permanent appearance of the piece.

The black-and-white image remained important to me. It gives the flock a clear silhouette and makes changes in density easy to see. It also gives color somewhere to arrive from. If everything is constantly colorful and intense, an additional effect has less impact. I wanted room for quieter moments, empty space, and gradual changes before a sudden visual accent.

The technical process involved several corrections. At one point, MIDI values were changing correctly, but the birds were not flying because the rotation export needed to be reconnected. Another attempt produced distortion without the color effect I wanted. I rebuilt that section around an explicit color mapping instead. These moments taught me to check each stage separately: controller input, numerical range, animation, and final display.

During the live performance, I was much more nervous than I had expected. Once the music started, though, I focused on listening and working the controller. I hit each drum beat with the visual changes I intended, and the controls worked smoothly throughout. The result felt even better than I had hoped. What surprised me most was how different the project felt when I was performing it: the birds were no longer just an animation I had built, but something I could play. Getting those moments right made the time spent adjusting and troubleshooting the controls feel worthwhile.

The most interesting shift was from watching a system to making decisions within it. The controller gave me a way to shape tension through accumulation, speed, distance, and interruption. For a future version, I would add clearer physical labels, a few reliable scene presets, and smoother transitions between them. I would also spend more time rehearsing when to leave the image alone. Building more effects is tempting, but learning when a small movement is enough is part of learning to perform this instrument.

Tools and Credits

• Software: TouchDesigner.
• Controller: Korg nanoKONTROL2.
• Bird assets: Model from my personal asset library, with rigging and flight poses prepared by me.

Groove Machine – Pressure Project 3

For my visual synth, I wanted to create two compositions that represented different vibes. My intention was to have the visuals begin slowly, transforming into a more intense visual scene as the song goes on. I was inspired by rave visuals, which have a very chaotic, colorful nature. 

The first patch is the simplest of the two, and is made up of a SOP feedback loop. First, I created a simple SOP composition using a copy node. I then ran this through a feedback loop that changed various transform parameters and some lighting adjustments. This allowed for quick feedback loop adjustment to create hypnotic, circular patterns. 

The second patch is a little more complex, and uses external images along with 3D shapes. The SOP composition is pretty similar to the first patch, but it utilizes 3D shapes, while also giving the ability for the user to switch between a circle and box shape. I wanted to make this patch more colorful, so I brought in some Photoshop designs I had done in the past and hooked them up to a switch, which cycles through the images depending on the speed of the LFO that controls the switch. These designs served as the background for the patch, but I also used them as textures for my SOP composition, which added more color and visual interest. 

After I finished my two main patches, my primary goal was focused on being able to switch between the two in an efficient manner. I was not aware that when adding a Geo, Light, and Camera, that every SOP in view gets rendered. This was not ideal when separating my two patches, so I hooked up my SOPs and final outputs to a switch, which allowed for MIDI control of both switches simultaneously. Further, there was some nice interplay between the two patches due to the feedback loops I used, which helped to blend together the final output. 

One thing that helped me with the live performance aspect was the use of LFOs rather than direct MIDI controls. During the in class demo, I realized that controlling every parameter through MIDI was not ideal as I was only able to turn a certain amount of knobs and sliders at a time. To make dynamic visuals that always had movement, I controlled most of my parameters with LFOs to ensure constant movement. For my MIDI, I mapped my controls to the speed of the LFOs rather than the parameters themselves, which allowed for easier, more efficient control. An example where this was most effective was through isolating parts of the song. Rather than continuously pulsing a parameter to emulate the bass or melody, I was able to beatmatch the parameter by changing the LFO, which let me focus on changing other parameters. 

You can see my full patch and screenshots using the link

The Integration of Traditional Culture and Modern Technology

For Pressure Project 2 — A Mystery Is Revealed, I created an interactive experience that combines traditional Chinese shadow puppetry with a story from Journey to the West. My inspiration came from the classic episode Sun Wukong （The Monkey King）and the White Bone Demon. By combining traditional Chinese storytelling and art with modern interactive technology, I wanted the audience to experience the fun of controlling shadow puppets themselves rather than simply watching a performance.

My project focuses on revealing the hidden identity of a mysterious woman. At the beginning of the experience, the audience first sees a landscape inspired by traditional Chinese art. A mysterious woman then appears, followed by Sun Wukong, the Monkey King. The audience can control Sun Wukong and try to uncover the woman’s secret. At the same time, another participant can control the mysterious woman, so the experience can be played by either one or two people. When Sun Wukong’s golden staff strikes the woman’s head, her human appearance gradually breaks apart, revealing the skeletal form hidden beneath her disguise. At that moment, the audience discovers that the seemingly ordinary woman is actually the White Bone Demon, and the mystery is finally revealed.

Process and Challenges

During the development process, the first problem I needed to solve was how to make the static shadow puppet characters move. To recreate the movable joints of traditional Chinese shadow puppets, I separated Sun Wukong’s character image into different parts, including his head, body, two arms, and two legs. I then imported these parts into TouchDesigner and used Transform TOPs to adjust their positions, sizes, and pivot points before putting them back together as a complete character. This step was more complicated than I originally expected. If the pivot points were not positioned correctly, the arms and legs would separate from the body when they rotated. I spent quite a lot of time making adjustments until the character’s movements looked more natural.

After assembling the character, I used MediaPipe hand tracking to create the interaction. Participants can control Sun Wukong with their left hand and the mysterious woman with their right hand. By bending different fingers, they can also control Sun Wukong’s head, arms, and legs. Both characters can move around the screen and automatically turn to face different directions based on hand movements. However, I encountered some unexpected challenges while developing these interactions. First, the hand-tracking system was very sensitive to small finger movements. Even when I tried to keep my fingers still, the character’s joints would continue shaking. Another problem was that MediaPipe did not always detect the two hands in the same order, which sometimes caused the characters controlled by the left and right hands to get mixed up. To solve these problems, I adjusted the hand recognition logic and used Lag CHOPs to smooth the characters’ movements, making the controls more stable.

Another major challenge was detecting collisions between Sun Wukong’s golden staff and the White Bone Demon. At first, I tried to detect attacks based on the distance between the characters and the rotation angle of Sun Wukong’s arm. However, this method was not very accurate. Sometimes, simply raising Sun Wukong’s arm would trigger the transformation, even when the staff had not actually touched the woman. To fix this, I redesigned the collision detection system by creating separate collision areas for the tip of the golden staff and the woman’s head. The transformation is now triggered only when these two areas overlap. When the staff successfully hits her head, her human appearance gradually distorts, revealing the skeletal form hidden underneath.

Through this project, I realized that interaction design is not just about making a computer recognize human movements. More importantly, the system needs to respond to those movements in a clear and meaningful way. I also learned that even small details can have a big impact on the user experience. Because of this, I had to keep testing and adjusting my project to reduce errors and make the interaction feel smoother.

Another thing I had to think about was how to make a simple character transformation feel like a moment when a mystery is truly revealed, rather than just another animation effect. To achieve this, I added opening subtitles, a question about the mysterious woman’s identity, and background music. I wanted these elements to create a sense of mystery and make the audience curious at the beginning, so they could eventually discover the truth through their own actions.

In traditional Chinese shadow puppetry, the audience usually watches performers control the puppets. However, in my project, the audience can become the puppeteers themselves. I think this change from watching to actively participating shows how modern interactive technology can bring a new kind of experience to traditional art forms. At the same time, I hope my project can encourage people who are unfamiliar with Journey to the West to become interested in Chinese traditional stories and culture, and perhaps learn more about them.