Cycle 2: Video-Bop
Posted: May 1, 2024 Filed under: Uncategorized | Tags: Cycle 2 Leave a comment »Feedback from cycle I directed me towards increased audience interactivity with the video-bop experience.
Continuing in the spirit of Kerouac, I was inspired by one of his recordings called American Haikus in which he riffs back and forth with tenor saxophone player Zoot Sims. Kerouac, not being a traditional musical instrumentalist (per say), recites his version of American Haiku’s in call and response with Zoot’s improvisations:
“The American Haiku is not exactly the Japanese Haiku. The Japanese Haiku is strictly disciplined to seventeen syllables but since the language structure is different I don’t think American Haikus (short three-line poems intended to be completely packed with Void of Whole) should worry about syllables because American speech is something again… bursting to pop. Above all, a Haiku must be very simple and free of all poetic trickery and make a little picture and yet be as airy and graceful as a Vivaldi Pastorella.” (Kerouac, Book of Haikus)
Kerouac’s artistic choice to speak simple, yet visually oriented ‘haikus’, allows him inhabit and influence the abstract sonic space of group improvised jazz. These haikus are at par with the music motifs typical of trading which is when the members of jazz ensemble will take turns improvising over small musical groupings within the form of the tune they are currently playing. What I find most cool is how you can feel Zoot representing Kerouac’s visual ideas in sound and in real-time. In this way, a mental visual landscape is more directly shaped by merging musical expression and the higher cognitive layer of spoken language. It is not new for abstract music to be given visual qualities. Jazz pianist Bill Evans described the prolific ‘Kind of Blue’ record as akin to “Japanese visual art in which the artist is forced to be spontaneous…paint[ing] on a thin stretched parchment.. in such a direct way that deliberation cannot interfere”(Evans, Kind of Blue Liner notes 1959)
As a new media artist, I tried to create a media system which would engage the audience in this creative process of translating abstract ideas of one form into another. I believe this practice can help externalize the mental processes of rapid free association. To do so, I had to build a web-application accessible to the audience connected to a cloud database which could be queried from my PC running the Isadora media software. This web-app could handle the requests of multiple users from their phones or other smart-devices with internet access. I used a framework familiar to me using Dash to code the dynamic website, Heroku to host the public server, and AstraDB to store and recall audience generated data.
See src code for design details
The experience stared with the audience scanning a QR code to access the website, effectively tuning their phones into an interactive control surface. Next, they were instructed to read Kerouac’s Essentials of Spontaneous Prose which humorously served as the ‘rules’ for the experience. This was more of a mood setting device to frame the audience to think about creativity from this spontaneous image oriented angle.
Next, I played a series of Kerouac’s Haikus and instructed the audience to visit a page on the site where they could jot their mental events for each haiku as they listened to the spoken words and musical interpretation by Zoot. After this, there was a submit button which sent all their words to the database and was then dynamically rendered onto the next page called ‘collective thoughts’. This allowed everyone in the audience to anonymously see each other’s free associations.
Example from our demo
After, reading through the collective image-sketches from the group, we decided on a crowd favorite Haiku to be visualized. The visualization process was equipped to handle multiple Youtube video links with timecode information to align with the time occurrence of spoken words in a prerecorded video. This process followed form from Cycle I in how I quickly explored Youtube to gather imagery which I thought expressive of the message within the ‘History of Bop’ Poem. This practices forces a negotiation in expression between original image thoughts and the available videos on the medium of Youtube equipped with its database of uploaded content and recommender systems. An added benefit to having the interaction on their personal phones is that it connects to their existing Youtube app and any behavioral history made prior to entering into the experience. The page to add media looked like this:
This was the final step and it allowed tables to be generated within the cloud database which were in a form for which they could be post-processed into a json file which worked with the visualizing patch I made in Isadora for Cycle 1. I had written a python script to query the database and download all of the media artifacts and place them into the proper format.
Unfortunately, I didn’t have much time to test the media system prior to presentation day and the database was overwritten due to a design issue. Someone had submitted a blank form which overwrote all of the youtube data entered by the other audience members. For this reason, I was not able to display the final product of their efforts. Yet, it was a good point of failure for improvement in cycle 3. The audience was disappointed that they didn’t see the end product, but I took this as a sign that the experience created an anticipation which had powerful buildup.
Cycle I: History of Bop
Posted: April 10, 2024 Filed under: Uncategorized | Tags: cycle 1 Leave a comment »Video Bop interpretation of an Excerpt from Jack Kerouac’s ‘History of Bop’
History Of Bop The History of Bop by Jack Kerouac
As a jazz enthusiast and young adult, reading Kerouac’s ‘On The Road’ was a transformative experience. In school, I was predominantly interested in the math and sciences and hardly cared to read a book or pick up a pen to write. However, Kerouac’s style, legacy, and approach to writing (and life for that matter) convinced me of the value in these types of intellectual pursuits. Over the last five years or so, I’ve continued to explore Kerouac and other works from the beat canon; One exciting find was his set of spoken word readings called ‘On the Beat Generation’. For cycle I, I focused on the final minute and a half section from his work ‘History of Bop’. I find this writing to be a triumphant portrayal of the evolution of bebop and cultural changes in America surrounding the genre.
Upon researching the piece, it was interesting to find that it was originally published in the April 1959 issue of an lewd magazine called Escapade – a hidden gem of writing amongst smutty caricatures and 50s advertisements. Although, I was not entirely surprised by this discordant arrangement given that the hero of the beats (Neal Cassady) was, according to Allen Ginsberg, an “Adonis of Denver—joy to the memory of his innumerable lays of girls in empty lots & diner backyards, moviehouses’ rickety rows, on mountaintops in caves or with gaunt waitresses in familiar roadside lonely petticoat upliftings & especially secret gas-station solipsisms of johns, & hometown alleys too”(Howl, 1956). This quality of the beats did not age well especially from the vantage of sexual equality, but unconventional behavior and criticism are the norm for this eclectic group. What I’d call a most foundational criticism of Kerouac and this piece of writing in particular is in the realm of black appropriation. Scholars like James Baldwin described this ‘untroubled tribute to youthful spontaneity [as] a double disservice—to the black Americans who were assumed to embody its spirit of spontaneity and to Kerouac’s full literary achievement… a romantic appraisal of black inner vitality’(Scott Saul, FREEDOM IS, FREEDOM AIN’T, pg 56). It cannot be denied that Kerouac and his writer friends were escaping what they feared as the trap of the middle class white picket fence, and couldn’t have experienced the true reality of being a black in the mid 20th century. Yet, their works speak to a deep respect and profound inspiration for the black art of their time (jazz).
One artistic technique embodied by the beats and inspired from jazz is what Kerouac calls ‘spontaneous prose’. Jazz musicians exceled at this rapid invention of musical structures through heightened sonic sensibility and borderline phantasmal prostheses of an instrument to their nervous systems and soul for that matter. The beats took up this approach with the technology of their time; namely the typewriter. Kerouac even made a manifesto called ESSENTIALS OF SPONTANEOUS PROSE in which he describes the procedure of writing as ‘the essence in the purity of speech, sketching language is undisturbed flow from the mind of personal secret idea-words, blowing (as per jazz musician) on subject of image.’ Other writers of his time, such as Truman Capote, didn’t see eye to eye with this artistic style and humorously commented “That’s not writing, that’s typing.” While I don’t mean to argue that thoroughly edited and fully composed art is better or worse than spontaneous artforms like jazz and bop-prose, it is just that spontaneous modes of creation akin to ‘play’ can potentially more honestly externalize such private and elusive inner processes occurring in the lawless relational environment of the mind.
It has been a good 70 years since these creative practices surfaced within the avant-garde art scene, and the technologies at society’s disposal for externalizing thought have tremendously improved especially due to the massive sea of networked image and video objects available to the average internet user. Further, a concentrated development of skills in computer programming may be analogous to the level of discipline in harnessing artistic technics like the saxophone, piano, pencil, paper, typewriter, voice recorder and so on. With this long winded explanation in mind, these ideas are much the backbone of my inspirations in new media art and what I wish to explore in this 3 cycled project.
In cycle 1, I set out to visualize the final section of history of bop. I’ve listened to this poem recited countless times and did a lyrical transcription – meaning I listened to the recording and wrote out all of its words. This is a common practice among jazz musicians and writers since as it helps internalize language. Next, I followed Dmytro Nikolaiev’s implementation of a Vosk LLM in Python to convert the audio file into a transcribed text JSON file which contained the words and their position in time as spoken.
Not all the words were accurately transcribed so I had to manually correct as AI models are good but not nuanced enough to decipher all spoken phrases especially from an unconventional speaker like Kerouac. Further, I did some post-processing of the data to get it in a form which would play nicely with Isadora’s JSON parser actor. The format involved Key-Value pairs of timestamps and words spoken. Through experimentation, I found that repeating each word in the JSON list near the millisecond (ms) frequency ensured that it would appear onscreen and remain illuminated consistently as the corresponding audio is spoken. I found that although the resolution of the timecode variable of the movie player actor was at the ms scale, it didn’t increment consistently enough to predict the values it would trigger. Consequently, having a large and widespread array of timestamps between the start and end of a word ensures that it will be triggered. Additionally, Isadora plays in the realm of percentages between 0 and 100 rather than the typical format of videos being in time duration. So I had to account for a conversion of timestamp as percent completed with respect to the total length of the audio clip.
This allowed me to funnel the current position of a playing audio file through to the JSON parser actor, such that as the timecode increments, the transcribed text would display onscreen exactly in time with the recited poetry. This was exciting on its own as it was a semi-autonomous method to generating lyric videos. Also, the style of the text was strobe like giving the quality of spoken words – they appear and vanish in an instant. See below the media circuit implemented in Isadora (with image player disabled) to see the flow of time triggering text and subsequently displaying on screen.
The next stage in the process was to find imagery representing the ideas that Kerouac is expressing. Following Kerouac’s ‘Setup’ step: ‘The object is set before the mind, either in reality. as in sketching (before a landscape or teacup or old face) or is set in the memory wherein it becomes the sketching from memory of a definite image-object.’, I used Kerouac’s speaking of the words to serve as the stimulus (object) for mental imagery. Once an image or set of ideas was established in mind through free-association (‘mental image blowing’), I would search Youtube to find a clip which best represented this mental image. Instead of using one of the many available and ad-prone sites for converting Youtube videos to .mp4 files, I adapted a Python script using the yt-dlp/yt-dlp library to do so with better speed and precision. This allowed me to quickly find sections of videos, copy their video url along with start / end time into a function which would download the video file with a specific name to a designated folder. This method allows more mental energy to be concentrated on thinking of images and finding existing internet representations rather than downloading and cutting the video segments. In this way a quick flow can be achieved, better mimicking Kerouac’s spontaneous prose method. To add, just as writing is a negotiation between image thoughts and the language available to one’s tongue and fingers at that moment, video-bop (new term for this method) is a negotiation between a visualized animation and the medium of available images / videos online. This medium includes not only the content available and the form they take, but also the algorithmic recommender process personalized by the user’s previous internet activity. For in this intentionally fast paced creative process, one relies heavily on differentiating search terms to approach on an appropriate visualization.
When videos were found, they were named with the first ‘semi-unique’ word within the phrase they belonged to, and the length of the video was chosen to match the duration of that phrase as calculable from the initial audio transcription step. The grouping of phrases is to the video-bopper’s discretion and in accordance to the aesthetic sensibility eminent in jazz’s musical structuring. It is not necessary to find video images in the order for which they are spoken. I’d hopped around between Kerouac’s phrases freely and would encourage this approach as it may follow the flow of thinking more closely and it builds natural structures of moments and transitions between moments. This idea was neatly phrased by Mark Turner, a modern tenor saxophone player who describes “When I’m in the middle of a solo, whenever I am most certain of the next note I have to play, the more possibilities open up for the notes that follow.”(The Jazz of Physics, Stephon Alexander). To riff on Heisenberg’s uncertainty principle, there is this interdependence on knowing the exactness of both a particle’s momentum and position. To extend this into the domain of thought and artistic expression, perhaps carelessly, it suggests that there is a tradeoff in awareness. When the improvisor’s awareness is tuned most closely to what idea should come next, they may be unaware of the larger artistic structure to emerge. In contrast, if the improvisor’s awareness is tuned to larger timespans and movements in the piece, they may have less awareness of the idea to come next.
As a continuation on methodology, Isadora unfortunately doesn’t read file-paths for media artifacts and rather relies on an internal numbering system as files are uploaded to the project. To accommodate this structure, I manually updated a JSON object to convert between video trigger words and their index in Isadora. These index values are passed into Isadora’s movie player object to allow them to be visualized in time with the typography and spoken words.
Upon showing the project to my classmates and delving into ideas on how it can be more interactive for cycle’s 2 and 3 to come, many important suggestions were accumulated to form the next direction:
- Jiang – Words that are action oriented may be good to include for transitions of images. (ie ‘turn’).
- Afure – She liked my interpretation of word ‘Dreaming’ – Although, everyone would have a different interpretation of that word and what img they’d select.
- Tik Tok Trend for going on Pinterest and searching words and displaying what image that is algorithmically connected with.
- Kiki – Liked the subject matter – she teaches jazz dance and it would be helpful to have a more interesting way to teach about the this genre.
- Alex – ways for interactivity – what sorts of ways to automate img generation and allow for user thoughts / personalities to be included. Potential for a custom web-app.
- Nathan – would like to have clicked on links related to the content being shown. As a way to learn more about each part (as informative).
With these suggestions in mind, I plan to explore the use of Dispatcher — python-osc 1.7.1 documentation to build a simple server hosted web interface open to smart-device users connected to a LAN. A spoken word poem should be found and disseminated to each of the audience’s device surfaces. As an experience, there should be a listening period in which the audience engages with their own forms of active imagination to see what phrases catch their ears and images that become naturally available to them. From there, they should go through the video-bop process and find a clip which matches what they’ve concepted. Then they will connect to the media server and paste the link of that video, the start and end time, as well as the phrase it connects to. The media server will need to collect these audience responses and run the Youtube extraction script to grab all the associated artifacts and make them available for rendering in time with the spoken word poetry. This is the direction I envision for cycle two and a diagram of how I see the interaction occurring:
PP3 – Scarcity & Abundance
Posted: March 4, 2024 Filed under: Uncategorized | Tags: Pressure Project Leave a comment »This project began with the idea to use a snare drum as a control event in relation to the Makey Makey device and Isadora. With the prompt of ‘a surprise is revealed’, I wanted to explore user experiences involving sharp contrasts in perspective through two parallel narratives. One narrative (‘clean path’) contains imagery suggestive of positive experiences within a culture of abundant resources. The second narrative (‘noisy path’) includes imagery often filtered out of mainline consciousness as it is a failure byproduct within the cultures of abundant resources. By striking the snare drum in accordance with an audible metronome, the user traverses a deliberate set of media objects. In photo based scenes, there is a continuum of images scaled from most to least ‘noisy’. Hitting the drum in synchronous with the metronome will enhance the apparent cleanliness of the image. While hitting off beat renders the reverse effect giving more noisy or distasteful images. In video based scenes, two videos were chosen which illustrate opposing viewpoints and similarly, the timing of drum beats alters the display of positive or negative imagery.
The first images are of random noise added to a sinusoid. A Python script was written to generate these images and an array of noise thresholds were selected to cover the variation from a pure sinusoid to absolute random noise. This serves as a symbol for the entire piece as Fourier mathematics form the basis for electrical communication systems. This theory supports that all analog and digital waveforms can be characterized by sums of sinusoids of varying frequency and amplitude. As such, all digital information (video, image, audio…) can be represented digitally in the form of these sinusoids for efficient capture, transmission, and reception. In the digital domain, often unwanted signal artifacts are captured during this process, so digital signal processing (filtering) mechanisms are incorporated to clean up the signal content. Just as filtering adjusts signal content from the binary level of media objects, it exists at higher computing levels, most notably algorithmic filtering in search engine recommendation, social media feeds, and spam detection in email inbox to name a few. Further, on the human cognitive level, societies with abundance of resources may be subjected to the filtering out of undesirable realities.
One such undesirable reality is that of consumer waste which is conceptually filtered through the out of sight, out of mind tactic of landfills. Even when in sight, such as in a litter prevalent city, the trash may be filtered out cognitively just as is done by audible ambient noise. The grocery store shopping and landfill scenes serve to illustrate this concept. The timelapse style emphasizes the speed and mechanicalness at which the actions of buying colorful rectangular food items and compressing massive trash piles occur. Within the system of food consumption, this video pair the familiar experience of filling up a cart at the grocery store with the unfamiliar afterthought of where those consumables are disposed.
The third scene takes influence from Edward Burtynsky’s photography of shipbreaking in countries like Pakistan, India, and Bangladesh in which a majority of the worlds ships and oil tankers are beached for material stripping. These countries are not abundant in metal mines and their economies are dependent on reception of ships for recyclability of iron and steel. The working conditions are highly dangerous involving toxic material exposure and regular demolition of heavy equipment which dehumanize workers and cause environmental damage. This video is contrasted by an advertisement for Carnival cruise line and a family enjoying the luxury of vacationing at sea on a massive boat containing a waterpark and small rollercoaster. This scene reveals an excessive leisure experience available to those in areas of abundance and the disposal process when these ships are no longer of use. Further emphasizing the sentiment of ‘what is one man’s trash is another man’s treasure’.
The last scene includes a series of screenshots taken from Adobe photoshop showing the digital transformation of a pregnant woman into a slim figure. Considering the human body as having an ideal form, akin to that of the pure sinusoid, manual and automated photoshop tools provide ways to ‘cleanup’ individual appearance to a desired form. With body image insecurity, obesity, and prevalence of cosmetic surgeries pervading the social consciousness of abundance societies, this scene registers the ease at which these problems can be filtered away.
The choice of the drum and metronome as a control interface is designed to reconstruct the role that conformity plays in decision making and exposure to alternate perspectives. It is suspected that most users will hit the drum on beat because that is what sounds appealing and natural. Here the metronome represents the social systems underpinning our formation of narratives around consumption and self identity. With the design of this media system, it is possible for the user to only experience positive imagery so long as they strike in phase with the metronome. But for those daring to go against the grain and strike off beat, they are greeted by a multitude of undesirable realities. It is my hope that in participating with this media system, that users realize the role that digital systems play in shaping perceptions as well as how our the style of our interactions alter the possibilities on what can be seen.
Resources used in this project:
- Makey Makey
- Snare Drum
- Copper Tape
- Drum Stick
- Isadora
- 10 Hours of time
Media Artifacts Used:
PP2 | Otherworld Luminous Mushroom Garden
Posted: February 5, 2024 Filed under: Uncategorized Leave a comment »Of the numerous interactive environments at Otherworld, the exhibit which caught my interest most was what I interpreted to be a mushroom garden. This occurs prior to entering the exhibit with the mechanical feeding room for the bull with pyramid shaped utters and groans at you when you enter the wrong feeding sequence. This room consists of an elegant three starred archway with three groupings of mushroom capped surfaces sitting above planters. All of these illuminated surfaces are animated by overhead projectors carefully placed, angled and programmed to giving the environment a feeling that it is alive and in motion. This is not unique as most of the exhibits at otherworld rely on projection mapping techniques; however, what stood out to me was its interactivity.
As demonstrated in the videos below, there are spinning colored discs mapped outside each mushroom cap, which when stepped on, cause a flood of color to spill outwards onto the floor. The structure reminds more of gas diffusion such as smoke, flames, or fog due to the speed of expansion, geometric structure and color. Although I could not tell precisely how the system worked, I imagine there was some element of camera or other sensing which could detect the area of color detected on those spinning disks. When your foot covers the disc, it triggers the color to flow – so I presume the system has a threshold at which it detects shadows present on the disc spot. Further, each flow pattern was unique meaning that no two times that you stepped on the disk would there be the same structure / flow observed. To me this means there must be some random variable detected from the environment which governs how the light will flow. I am familiar with these types of flow generation techniques such as this reaction-diffusion generator made by Karl Sims https://www.karlsims.com/rdtool.html based on mathematical models of physical phenomena https://en.wikipedia.org/wiki/Reaction%E2%80%93diffusion_system. While they probably don’t have as complex of a mathematical model as described above, it still serves to suggest that there is some math involved to compute how the light will flow across a surface based on some initial random variable.
As a study in natural interaction, I observed several people as they entered the room and played around. For most, they seemed impressed by the scenery and were engaged visually by the Moire like pattens visible on the mushrooms. I could tell this by how the setting drew in their attention. However, most people didn’t notice the interactive element as they didn’t step on the spinning light discs. There was one person who did realize the hidden interactivity and chose to step on all of the discs. I found it interesting that she chose to jump from one disc to the next as they were spaced within hopping distance. This reminded me of games I’d play with my sister as a child which we called the floor is lava and we would try to move through a room only stepping on furniture and not the carpeting. As a benevolent alteration to this interactive design, it’d be interesting to account for tracking of human movement from one disc to the next. Perhaps if the person can jump from one end of the room to the next while triggering each of the discs, that the room will acknowledge this accomplishment as is done in the next room when the right sequence is fed to the bull utters. This may be technically challenging given the short detection time allowed for a sensor when someone is hopping quickly through an environment. Secondly, there would be a challenge in synchronizing detection of events from multiple cameras; yet, I bet there is already a centralized media server acting as the master to its various child projectors and sensors.
Pressure Project 1 | Neon Fragmentation
Posted: January 30, 2024 Filed under: Uncategorized Leave a comment »This piece was inspired by my personal interests in jazz, feedback systems, and media archeology. I’ve played saxophone for 10+ years and look highly upon jazz musicians as fearless explorers of sonic harmony, melody, and rhythm. Their artform requires a fine-tuned motor pathway system connecting the brain and fingers to enable quick musical decisions within an improvisatory ensemble of horns, drums, pianos, and strings. The interactions between musicians in an ensemble are unique and require a auditory feedback to create an artform bigger than any one contributor. In addition to jazz, I’ve worked with electronic feedback systems and learned to respect the impact their discovery has had on much of modern technology. These technological systems form an ecological relationship with human culture which too is receptive to feedback. The process of media archeology can help to scrutinize and analyze dominant narratives about technology and their relationship to culture.
For this project, a computational multimedia software called Isadora was used to devise an audiovisual feedback loop. This system inputs a video file undergoing several stages of video mixing and feature analysis to synchronize low and high frequency audio tone generation with video splitting. Computer vision plugins were used to detect relative motion and color within the inputted video frame. The intensity of motion detected was used to drive the quantity and frequency of video splitting in the XY directions. Additionally, the RGB color proportionality drove emission of audio tones where red produced frequencies between 30 – 400 Hz and green and blue produced frequencies between 1-2kHz.
This piece takes influence from early 1920s Montage film technique including Fernand Leger’s Ballet Mecanique. The affects of the industrial revolution are visualized through mechanical systems held at the same level of authorship as humans. Humans are depicted with cyclic motions such as a woman on a swing evoking the image of a pendulum. The rapid frame changes are also accompanied by a frenetic sound track which is synchronized to the rhythm of industry. The rapid montage clips remind me of jazz because of its quick delivery of stimulus transcending the traditional mental faculties of comprehension. It seems to me that Leger is playing around with visual Jazz in the form of semiotics. When passed through my generative media system, the quick color and shape changes lend themselves well when converted into high and low frequency tones. To the right set of ears, its difficult to tell the difference between abstract jazz and machine noises.
The assembly line style of interaction between humans and machines during and after WWII, is then expressed by a 1950s supermarket film. The women in the store is surrounded by an array of homogenous consumables which funnel smoothly down a conveyor belt, into her car, and then presumably into the home. There is an element of time ticking in the store from the machine noise pressing of keys on the cash-register to the conducting of cars via neon traffic light systems. This exchange of information and household goods all flows to the beat of machine inputs and outputs.
Human-machine stimulus and interaction is even more pronounced in the situation of the 1970s Las Vegas strip. Suburban tourists have gathered in neon wild west in search of cowboys and gold nuggets. Immersed in an ecosystem of electric light and casinos, tourists can at a safe distance engage in a symbolic exchange of money and neon. Grande advertising spectacles erected in the desert now being closer in association to sinful redlight districts than with their initial connotation of technological optimism.
I hope that this piece brings attention to interrelations of human and machines in the 20th century. The aesthetic of multivid splitting by computer vision and foreign sounds helps bring these media artifacts into the vantage of the modern algorithmic viewer. Just as machines influenced human perception and behavior, so too are algorithms.
Media artifacts:
Bump: Pressure Project #2: Dancing Depths
Posted: January 23, 2024 Filed under: Uncategorized Leave a comment »I find this project to be technically stimulating and conveys an interesting message about humans. Familiar in anatomy, but there is almost a chameleon like skin tapering around giving an alien / post-human effect. It seems that Axel used a kinect device which I know uses infrared technology to track movement. In this way, the computer vision filters out the projected light spectrum which is interesting. As opposed to the way we see the these ethereal neon figures, the computer just sees boundaries. Meaning the camera’s are unaffected by these artificial skins or personas mapped onto humans. It makes sense that this IR tech was chosen because I imagine it to be more accurate in low light conditions and less computationally expensive than visible light computer vision algorithms.
However, I still think it interesting to experiment with this project using a camera or a mix of IR and visible light processing. This allows for the interactive system to experience humans in both their anatomy and artificial skin. From a psychological level, this tradeoff in accuracy speaks to confusion between what we are and what we want to be. I think a bit of messiness in the tracking human motion and projecting onto account for the unclean boundaries between the two. The camera will have to contend with analyzing both the accentuated projected light and its faint shadow of physical form.