{"id":5840,"date":"2026-05-05T22:38:25","date_gmt":"2026-05-06T02:38:25","guid":{"rendered":"https:\/\/dems.asc.ohio-state.edu\/?p=5840"},"modified":"2026-05-05T22:38:25","modified_gmt":"2026-05-06T02:38:25","slug":"cycle-two-a-3d-movement-based-sound-explorer","status":"publish","type":"post","link":"https:\/\/dems.asc.ohio-state.edu\/?p=5840","title":{"rendered":"Cycle Two: A 3D Movement-Based Sound Explorer"},"content":{"rendered":"\n

Going into cycle two, I immediately knew that I wanted to take the ideas I had done in 2 axes of movement in cycle one<\/a> and translate it to 3D. In order to do this, I needed to rework the analysis portion to plot all the samples in 3 dimensions, as well as figure out a new way for the user to see their movement within the corpus of sounds. For several days, I tried to do the analysis section on my own, but found myself unable to figure out how to plot them in a 3D space as I’m not great with Jitter, Max’s visual package. However, just about when I was getting ready to give up, I found someone who had already done exactly what I was trying to do<\/a>.<\/p>\n\n\n\n

\"\"
My saving grace this cycle<\/figcaption><\/figure>\n\n\n\n

I won’t get too in-depth on how this works. But in short, it reduces the MFCC analysis data to 3 dimensions, as opposed to 2 in cycle 1, unpacks all that data into a matrix with 3 dimensions, and then that matrix data is used to generate an OpenGL environment with all the samples plotted in a 3D space. Then, depending on whatever point the camera is closest to in the Jitter window, the sample that correlates with that point is played back. This camera, however, was driven with WASDQZ, so I had to implement a new system of control that allowed this 3D corpus explorer to interface with the Google Mediapipe data I was receiving from TouchDesigner. I did this by scaling the data to the 3D space, smoothing it out some, and sending it a as list into the position attribute of the jit.gl.camera object. <\/p>\n\n\n\n

\"\"
Routing of the MediaPipe data over OSC<\/figcaption><\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Zoom-out of the 3D corpus space, each dot represents a sample; dimmer dots are deeper on the Z-axis<\/p>\n\n\n\n

Results<\/h2>\n\n\n\n

Unfortunately, I don’t think my vision for this cycle translated to reality as well as I’d hoped it would. One of my biggest issues was that I left on an option that locked the camera to (0,0,0) in the space. This caused a major decorrelation between the movements of anyone using it and the movement in the space. Everyone seemed to struggle with “navigating” the corpus space much more than in the 2D iteration. Once I got rid of this attribute during our discussion afterwards, people understood what was happening better right away. I think the camera was also just a little glitchy since MediaPipe isn’t the *best* at detecting depth, which caused it to jump around wildly. I think if I were to continue with this 3D idea, I would have to incorporate a depth sensor in some way so that all 3 dimensions are accurate. <\/p>\n\n\n\n

In my first cycle, I also included the MediaPipe landmark camera feed in the projector image as a sort of monitor for movement. In this one, I made the decision not to include it, as I’d hoped the movement in a 3D space would translate more intuitively to the user. However, I was really surprised that most everyone actually missed its presence. I think the playfulness of the little dot stickman moving around on screen must have provided a nice counter to the cacophony of sound. For this reason, I made a point to include the landmark skeleton in the final version. <\/p>\n\n\n\n