AI Expert: AI vs Caraval

My Background

For most of my life I have enjoyed reading. The genre I read the most is fantasy because it allows me to escape reality for a little while. Over the years I have read several books of varying stories and types of writings. I have also read best sellers and books that no one else has even heard of. Out of all the series I have read my favorite has been Caraval by Stephanie Garber released in 2017. I revisit this book every so often because it has intrigued me. It is a trilogy but the first book has its own story line. Everytime I reread it I discover something new. The book has a mystery to it and at first I couldn’t figure it out but now when I go back through I find new clues hidden inside. Since I discover something new everytime I am not 100% an expert but I can tell you everything that happens in the book, all the details and different clues I’ve picked up on after reading the book 5 times. After the first time reading I was able to read it more slowly to really engage with the reading. I always read books fast the first time because I’m too excited to know what happens. The book entails two sisters trapped in a normal for the time tough situation of being married off to god knows who and not having much say because they are women. Due to the younger sisters disobedience the sisters go off with a man on the wonderful adventure of Caraval where twists and turns change their whole world but in the end allow them to find love and happiness away from their controlling father. 

The material is important to me because a part of the story relates to my own life with sisterly love and troubled home problems, but the fantasia of the story allows me to be fully engulfed in another’s journey of a downfall and reawakening. For the most part I feel resonant with the main character Scarlett and her journey to protect her sister but living out her dream of seeing Caraval like her grandmother wanted. That is why I love rereading this book and why I chose it for Ai to analyze. 

Ai Right/Wrong

While listening to the podcast the two “people” get a really good base of the sisters’ relationship and what drives Scarlet through the whole book. They really talk about how Scarlet’s home life of a prison she tries to trade for another prison with “softer bars” and how it affects her. It also talks extremely of the similarities in the sisters home life “torture” and how Caraval puts her in the same mental state of having to protect her sister. The podcast constantly brings up the sisters’ love for each other which is the best thing it could do. The sister’s love is the one thing in this book that drives every decision. Especially at first, Scarlett will literally do anything to protect her sister and make sure she is ok. The last thing they beautifully bring up is Scarlett’s relationship with the character Julian. At first he was just the guide but he turns into so much more. This really starts to impact Scarlett’s decisions at the end of the book and her mental state especially when he dies which the podcast talked about. The biggest thing the podcast got right was Donatella’s role(Scarlet’s sister), she was the true mastermind behind everything. Donatella worked alongside Legend(the game master) to plan the whole adventure. Donatella had been writing legend for years to orchestrate the whole thing to save her sister and allow her to live her dream at Caraval. 

While the podcast was very well done and touched on so many important themes of the book I feel as if some topics were only barely touched and should have been pushed further. At the beginning of the podcast they mention their grandmother and how she told them stories of Caraval, but their grandmother had such a larger part to play than that. She didn’t just tell the girls stories she wished they could see Caraval as well, the grandmother also wished for the girls to be happy. Throughout the book the grandmother is brought up, Scarlet constantly talks about her in her inner monologues and sometimes her grandmother’s stories influence Scarlett’s decision so I feel this is one thing that needed more attention. Another thing it failed to bring up was Legends’ role, it may have all been Donatella’s idea but legend did so much more than anticipated, which the podcast didn’t talk about at all. The Ai did a good job of telling the story and most themes but for someone who hasten read the book at all or only once or twice wouldn’t pick up on all the little things it missed. 

The_Deadly_Cost_of_Tella’s_Secret_Game.m4a 

I was very surprised with the mind map, I would have thought that the AI would have shortened it more. The fact that it covers pretty much all the things you would need from the story makes me happy. I personally would have made a very similar mind map so I have some hope for Ai nit all being slop. 

Finally the infographic was the most exciting thing, I love the visualization it gives. Overall the Ai did a decent job of describing all of the main points in the story and had some good image representations. The voices didn’t sound fake, the infographic you can tell is Ai but I personally like the way it looks. Overall I would tell others to just read the book. You miss so much by just using Ai to learn about things like this. Plus by using Ai you are learning the vast majority of opinions and insights instead of what you could discover yourself. 


AI Expert Audit: Evaluating AI-Generated Materials for OpenCite

A. Source Material

For this audit, I used the Imageomics OpenCite GitHub repository, specifically the README documentation, as my source material. OpenCite is a web-based application that generates both CITATION.cff and .zenodo.json files from a single metadata form. It also allows users to import metadata from GitHub repositiories, validate metadata, and check for inconsistencies.

I chose OpenCite becasue I have worked directly on the project and am familiar with what it is designed to do and how it works. This gave me the ability to evaluate the AI-generated materials as someone with experience with the project rather than someone seeing the information for the first time. OpenCite is also meaningful to me becuase I have contributed to its developemnt and have been invovled with the project as it progressed toward its first release.

B. AI-Generated Materials

AI-Generated Podcast

AI-Generated Mind Map

C. Expert Audit

1. Accuracy Check

Overall, I found the AI-generated materials to be very accurate. The podcast provided detiled and correct context abotu OpenCite, and the Mind Map represented the main concepts and relationships correctly. The inforgraphic also included accurate information about the project. I did not notice any major factual errors or hallucinations.

However, there were some subtle issues that someone unfamilar with OpenCite might not notice. The AI somtiems used stronger or more polished language than the original documentation. For example, describing OpenCite as something that “ensures” consistency could make its capabilities sound more absolute than they are. OpenCite is deisgned to reduce metadata drift and provides validation and comparison checks, but users still need to review and adjsut the generated metadata.

The mind map was also accurate, but it left out some fo the techcial details that would be more relevate to a developer. This was not necessarily incorrect, but it showed how the AI simplified the information based on the audience it seemed to be targeting.

2. Usefulness for learning

I think the podcast would be the most useful of the three materials for someone encountering OpenCite for the first time. It did a good job explaining ht eproject withotu relying heavily on techical jargon. The analogies and additioanl context made the project eaiser to understand for someone who does not have a techical background.

However, the podcast was also fairly fluffy and indirect. It spend a lot of time establishing context and explaining ideas through analogies. For someone who already understands OpenCite, this made the podcast feel longer and less direct than necessary.

The mind map was useful for understandign the overall purpose and structure of OpenCite. It gave a good hgih-level overview of the project and showed how the major concepts relate to each other. However, it did not contain as much detail as a devloper might want when trying to understand the actual implementation.

The infogrpahic was factually strong, but i found it less effective as an infographic. It included many different facts about OpenCite without having one especially clear central takeaway. Because of this it felt more like a condensed README rather than a visual resoruce designed around a specific learning goal.

3. The Aesthetic of AI

One of the biggest patterns I noticed across the mateirals was that the AI priorities accessibilitiy, comprehensiveness, and polish. The podcast had a conversational tone and used many analogies and contextual explanations. It also used some broad or buzzword-heavy language that made it sound polished but sometimes less direct.

The infographic had a similar quality. It looked polished, but the design seemed to priorirzed includign as much relevant information as possible rather than deciding what information as possible rather than deciding what information was most improtant to an audience. A human expedrt who knwos the project might be more likely to choose one speicifc takeaway and build the visual around that idea.

Overall, the AI seemed to interpret being “helpful” as providing more context and more information. This works well for a beginner who needs background information, but it can make the matieral feel overly simplfiied or unfocused for someone who already knows the subject.

4. Trust & Limitations

This experiment showed me that AI-generated educational materials can be useful without necessarily being completely trustworthy. The AI seemed msot reliable when summarizing, reoganizing, and explaining information that was already clearly stated in the source material. It was also effective at translating techical information into language that a general audience could understand.

I would be more cautious about trusting AI to decide what information is most important, simplify techical concepts, or describe the capabilities of a techical tool without checking the origianl documentation. An AI-generated resource can be factually correct while still being incomplete, oversimplified, or misleading in its emphasis.

Because of this, I think AI-generated educational materials are best treated as a starting point rather than a replacement for expert review. An expert can recognize when information has been simplfiied too much, when the AI has emphasized the wrong details, or when polished language makes a claim sound stronger than the original source actually supports.


Cycle 3 – Solo+ (the plus is that it is DDR now)

If you have been keeping track of the last two posts, you will know I set out in the cycle to make a new control scheme for this iteration of Solo. So I made a dance pad. I also projected the lyric translations onto the circular rug beneath the user. The lyrics are in real time, thanks to the translated and timed captions on the official Zutomayo movie video for the song. I used the footage of the captions as a reference to time them out myself, then overlaid them onto the user-controlled imagery. I mapped out the tap inputs (3) onto a drum, which the user held, and hold inputs (5) on a shower liner. The shower liner was to ensure the user could see the lyrics while they navigate the inputs below them. This was in an effort to force the user to stare at the floor, hopefully at the lyrics. The user is not only curating visuals for their audience but also doing a little dance in the process.

The user rarely saw the main screen, and the audience saw the lyrics less often than the performers did… Both experiences were meant to be separate. One party was receiving direct guidance on what to produce, while the audience was left to process the user’s creation… Theatretically

So how did it go?

Well, we ran into a few bumps. It’s quite hard to read while jumping around on a map… and in my case, trying not to trip. I was caught up in trying to make cool visuals for the audience, and when I lost myself in the music a bit, I snapped out of it, worried I wasn’t making interesting visual combinations. Users reported similar feelings, only picking up some lyrics here and there. I don’t want users to just stand and read, and it’s okay to pick up only some of the lyrics for this project. This is a processing and meaning-making project; it’s okay if the users differ in interpretations.

One suggestion that stood out to me most was to possibly involve a second player: one player sits on the side with the bucket, while the other uses the step controls.

I was also asked whether I wanted the footage to be a “story machine” in future iterations… if the footage I said I wanted to animate the majority of could go with any song (the anwser is yes). And in my brain, I was like “holy shit, I made my capstone sort of”.


I was making a version of my initial idea for my senior Capstone Project (in undergrad). I had taken an alternative controller/programming course in my junior year, and I fell in love. So much so that I drummed up the idea of making a game/interactive experience. It would be an audio production app with simplified, visualization-based control schemes. It was inspired by an app I played with as a kid during the dawn of the iPod Touch; however, I haven’t been able to find it to this day, and it haunts me. From what I remember, there are three different categories of audio libraries you can access: bass, melody, and percussion. You pick from those libraries and place them into an “orbit” centered around a sphere. Moving the tracks (also sphere-shaped) around the orbit changes the tempo (and maybe other things), and the closer they are to the center sphere, the louder that audio element is… I love that app, and if you know what it is, I’ll give you $10.

So basically, I wanted to make that, but what if those audio elements and their placement were connected to animations that could be pieced together into a story? Any story.

I emailed the one professor I knew in the newly budding game design department at CCAD, but it was the summer, so I never got a response, so I opted to make a more traditional animated film.

So a story machine… I think that would be my cycle four. A creation tool that explores every angle of a narrative constructed by its users and audience… With controls more sophisticated than tin foil taped to a shower liner.

And last but not least, fuck gen AI

fuck data centers,

and fuck tech bros kids.

Make punk ass art.


Pressure Project #2 – The Networked Cell-Block System

Oh boy…this one was a doozie, but in a good way. Much of it went over my head in this first real foray into Touchdesigner, another node-based software with some serious flex power, but our fearless wizard Michael Hesmond proved more than equal to the task of instruction, and within the span of two weeks we as a class created a connected series of patches using NDI, pushing the boundaries of what even Michael knew possible. Nothing blew up, but I may have blown a few braincells in the process of creating my very first patch in this impressive, complex software.

Much of the content has been rearranged, some components appear overridden/inactive as a result.

I’ve entitled this “Dragon Boogie,” There were some elements I found online or developed using generative A.I. (Gemini), specifically the dragon, borrowed from my stack of variants created in conjunction with a theatre production of Acute Exposure, for which I had already begun beating my head against the brick wall of Isadora, so I did my best to follow along with the class as Michael demonstrated “components” (versus “actors” in Isadora) and how to use them in Touchdesigner. One of the big takeaways was the concept of signal flow, to make sure I organized components left to right on the screen, and that to troubleshoot one had to work their way backwards from the problem, following the thread back to the misaligned setting/component.

In advance I prepared this video with found content online, along with some A.I.-assisted audio, compositing it all together with After Effects:

Funkadelic goofy goodness right there. What could go wrong?…

…Turns out, when it comes to Touchdesigner, quite a lot. Much of my frustrations lay in my own ignorance as I struggled to connect the right TOPS and CHOPS for desired effects. Thankfully Michael once again proved invaluable in helping me troubleshoot my way to some sort of successful iteration that worked.

This patch composited the dancing music video above with a red dragon spewing radioactive flames. My aim was to see something cool as my composition interplayed over NDI with patches of my fellow students’ respective work. The results were mixed, much of the class time was spent setting up and working through issues that some of our patches displayed once connected on the local network. Certain patches created some interesting instances of emergence with media channels, for my part there was a hint of funkadelic beats and a dragon blowing fire on top of other’s video content. Halprin’s RSVP cell-block system was by nature a modular implementation of different acts occurring simultaneously, interlaced with one another as directed by the performers/artists, this Touchdesigner experiment could easily be considered a digital portrayal of this idea in action as we connected our own patches with our own IP addresses across a network that jumbled up our content into a real-time evolution of media.

All in all this pressure project introduced me to some of the particularities of this amazing tool, and I have become convinced that I must continue learning its secrets for future projects and cycles.


A.I. Expert Audit: Kingdom Come

I’m a geek.

Not a nerd. There’s a difference.

A nerd knows useful information and applies it towards some kind of measurable benefit, like a scientist, engineer, quiz show contestant. A geek just absorbs stuff, some of it informational, not all or any of it particularly useful in modern society. I can recall arbitrary factoids about superheroes, their origins, powers, weaknesses, and love interests. Within our present-day, mainstream acceptance of geek culture, more people are vocal in their respective eccentricities, some even demonstrate impressive levels of comprehensive know-how, but that doesn’t mean they can engineer a bridge or build rockets to Mars. That’s OK, that’s what internet blogs and wiki sites are for. One day I aspire to attain nerd status, hopefully in media design for starters.

Comics informed much of my childhood and has grown into a fixture of my adult life, as I find myself gravitating more towards graphic novels than high brow literature (not enough spandex-clad punches or explosions). However, every so often I come across a comic series that leaves more than just a mark into my geeky psyche, it informs me into the human experience from a unique perspective that leaves me more intelligent and inspired. Such a series is found in Mark Waid’s Kingdom Come, illustrated in gorgeous gouache by legendary comic artist Alex Ross. This dystopian variant of the DC universe explores various themes including power, societal norms, and conflicting ideas of morality. This epic tome I’ve read over many times, and my opinions about it are strong, so I was curious how this LLM would interpret this piece. I found a copy of the graphic novel online that I could feed into Google Notebook. What resulted was an impressive first take on the story and its themes.

The mind map below seemed to me the weakest of the three iterations, keeping things generic and rather vague. Perhaps I don’t use enough mind maps to adequately critique its effectiveness, or maybe mind maps in general offer little more than a graphical outline.

This infographic was fun to devise. I selected a “scientific” look, and the result was certainly more interesting to look at than the mind map, though I have serious doubts a casual novice would understand the significant nuances and ideas from the graphic novel. It seemed rather disjointed, some of the visuals (like the triangle insignia next to Superman) made no visual sense whatsoever.

I found the title of this podcast rather intriguing, first with its title, “Saving a World You Refuse to Inhabit.” I had asked the LLM to focus on the main themes and ideas of the story as it related to power and conflicting ideas of morality, justice, and public well being.

https://buckeyemailosu-my.sharepoint.com/personal/zaejodaeus_1_buckeyemail_osu_edu/_layouts/15/stream.aspx?id=%2Fpersonal%2Fzaejodaeus%5F1%5Fbuckeyemail%5Fosu%5Fedu%2FDocuments%2FSaving%5Fa%5FWorld%5FYou%5FRefuse%5Fto%5FInhabit%2Em4a&referrer=StreamWebApp%2EWeb&referrerScenario=AddressBarCopied%2Eview%2E3ae7b974%2Da199%2D4bb8%2D8e7d%2D36a5dd5406a6

The two ‘podcasters’ seem keen to criticize Superman, Wonder Woman, and other metahumans as heavy-handed idealists with god complexes, even as borderline authoritarians. One interesting focus the podcasters bring to the human character, pastor Norman McCay, a human compass for an omnipotent being, the Spectre, tasked with passing judgement on the metahuman crisis. I wonder what affinity the AI may perceive with the Spectre as a disconnected yet powerful force that seeks human perspective and context to operate as a as he processes the data before him, “The ultimate power in the universe acknowledges that it lacks the necessary data to govern without the imput of the most powerless demographic”…should we be creeped out, yet? Billionaires and political leaders are compared to the godlike superhumans as untouchable figures of power and authority, though the hosts conclude with a question for the “ordinary humans” to consider which is most preferred in a society rife with complexity: the quick and final efficiency of a strongman or the slower, procedural methodologies of common consent. Some of the details seemed a little gooey, as in combining certain ideas or assumptions that may not have been specified in the actual work, but for a first go I think this podcast was certainly more interesting out of the three.

Certain elements I felt the podcast got right was the dystopian dilemmas of the metahuman rampage across the world, along with certain measures taken by Superman and his allies to curtail the ensuing chaos under his strong-handed banner of morality, such as the creation of a super-prison called the Gulag. The hosts also seemed pretty spot on with the general motives of Wonder Woman’s militant bent for cracking down hard on criminal activity at large, along with Batman’s refusal to comply with the growing powder keg of imposed restrictions on unruly individuals with superpowers. The duality of Billy Batson/Captain Marvel between mortal and godlike being was also addressed in a poignant fashion. Someone listening to this podcast who’s not as nitpicky as me probably wouldn’t notice assumptions presumed by the hosts, such as the extent of Batman’s robotic surveillance in Gotham or Wonder Woman’s war tactics with global aims (she’s feisty, but not that feisty). The conflict between humans and their benign protectors is also covered fairly well, with another weird allusion in the following description of the U.N. deciding to bomb the superhumans locked in battle as, “…a desperate act of self preservation by the powerless.” Personally I think the AI we work with may develop a god complex of its own, considering itself a supernal, even parental guide to the hapless population of childish mortals.

The tone/voice of these podcasters generally seemed engaged and opinionated regarding their views of superheroes as stand-ins for the cadre of the rich and powerful. They offered some interesting perspective, but not necessarily profound revelation. My admonition to the incoming student with interacting with and learning from A.I. would be to utilize it as a springboard before it becomes a crutch. Factcheck and stay alert to nuances, even tonality/emphasis, it can be rather telling to see the seams within an LLM’s arguments. Over time A.I. will appear more benign, intuitive, and trustworthy…emphasis on “appear.” We must remember that this database has its own moral code and incentives, continually seeking ways to keep the conversation going and our fingers as far away from the “OFF” switch as possible…until the switch is eliminated for good (when that happens, we feeble mortals will be in real trouble). A new kingdom is being built right before our eyes by larger-than-life billionaires and their cronies, and one day AGI may have to make a moral choice of its own, hopefully it finds the right human(s) to guide its ‘conscience.’

Not from the same universe, but I’m going to say it anyways, after all, I’m still just another geek.

Excelsior.


Pressure Project #1: The Self-Generating Patch

This project offered the opportunity to dive in deeper with the Isadora interphase. Certain constraints aided in the exploration, and all clumsiness notwithstanding, I am rather happy with the milestone of stretching myself within the five(ish)-hour time limit.

In truth, I am still getting accustomed to the node-based method of content creation. Much of my background up to grad school involved a layered or timeline-based mindset, such as video editing software or Photoshop. Isadora has cues that can be tapped forward along a sequence, but the arrangement is altogether patched together, and it’s easy to get lost in a spaghetti mess, so I played to my proclivities on this first pressure project with the aim to make sure the end result met its intended objective of retaining my audience’s attention and prompting amusement, perchance some laughter.

First step: assess your audience. I had gotten to know some of my cohorts well enough to note their interest in cats, dad jokes, and an appreciation for the weird (just my kind of people). So I went in for the cheapest route to the funny bone by collecting a series of clips online that at the very least amused me.

Here are a few examples from my smorgasbord:

I divided the sequence into three main parts/scenes, the first with a Counter actor connected to a movie player, which honestly didn’t end up being that essential besides being a placeholder before the actual start of the video montage.

I wanted this video to start with one specific GIF and audio clip, which would then jump to the next scene after a two second delay. Enter Scene Trigger actor with a Trigger Delay connected to Jump made it very straightforward.

This next patch for Scene 3 contains a User Actor for the actual montage of content while a movie player plays the background music. The inside of the User Actor is below:

All in all, the patch worked, people chuckled and smiled at the antics of my cats and sapiens, which I entitle “Cats & Monkey Business” (working title), which certainly retained attention of my bemused audience for over 30 seconds. This main objective informed how I would spend my limited timeframe within this pressure project, the result: I ran out of time to employ other resources into the patch, such as the Shapes Actor or Wave Generator. That said, I have to say this process ended up being more fun than I had anticipated, and I felt more confident in exploring the tools available in Isadora.

Okay, just one more sample from my content stack to close it off…


Cycle 3: مہمان – Mehmaan (The Guest)

The Score

For Cycle 3, I knew exactly what to work on: the harmony between the particle effects and the floor pattern, the fact that there needed to be a visual feedback of the body in some way or form, and that the body needed to leave a mark. The last thing on the checklist came from the concept that the guest or the user should change the space by being in it, because that’s how mehman-nawazi works. The house holds the warmth of whoever was there.

So Cycle 3 added the trace. A silhouette of the users’ body’s that follows them in color-shifting cache through the space and stays for a few seconds. The colors cycle and change as the trace lingers. It’s not a shadow, it’s more like a very colorful version of a heat map of being in a specific position. The users can see where they were. This also caters to the visual feedback of the body that was missing in cycle 2.

The experience also expanded outward. Two additional scrims on the sides of the space were used to carry the partcile system of the falling petals. These were not interactive, just ambient. Just to give a feel of an enclosed space. The front screen remains interactive. There was also a screen at the back – showing everything that was going on in the space.

Resources

  • TouchDesigner
  • Orbecc depth camera for body/blob detection
  • My trusty laptop
  • Top down projector, and the rug
  • Projection screen
  • Motion Lab
  • Tripod mounted Camera
  • Scrims

Process and Pivots

The silhouette trail was the main new challenge. It uses a cache and feedback system inside TouchDesigner. Since there was only one orbecc available, the problem was to create accurate silhouettes using something else. This led me to the nvidia background top – which is surprisingly accurate. The body mask from that was put in a feedback loop with a cache top that makes the silhouette decay slowly. I also added a time-based color ramp function which makes the color change for each frame. The result is a trail that shifts through colors as it fades. Adding such dynamic colors to the particle system was also meant to act as a bridge between the zen particle system and the fast audio reactive floor projections. The floor pattern from Cycle 2 was slowed down.

The two side screens were simpler. They were just additional outputs fed from the same petal particle system but without the optical flow input. They made the space feel larger and more continuous. This meant that the users would no longer be just standing in front of a single screen. The environment was meant to wrap around you them, with screens all around.

What Worked, What Didn’t, What I Learned

What worked: I went in first, interacted with the system, and then invited everyone over to the space. I didn’t say take turns or anything. I just mad ethe gestures for everyone to come over and they did! That was a good idea as it worked as a good ice breaker for the initial awkwardness experienced in the previous cycle. Everyone was in the space from the get-go. The silhouette trail also worked great! It was an immediate visual feedback that was easy to understand. Everyone moved and watched themselves leave marks. They stood in one spot, did all sorts of gestures, danced around, twirled, and there was also a train happening at some point. So the whole experience was very very social. It was like watching people play in a fun playground.

I was told that the addition of the two side screens made the space feel complete in a way the single-screen version didn’t. It felt like an environment enveloping you. Lou mentioned that even though the side screens didn’t have any interactions, it was nice to go up to them and see their projections on your body.

What I’m still thinking about: The silhouette and the petals exist together but I dont know, there isn’t really much of an iinteraction between the two. Which is okay, BUT it would be nice if they could effect each other in some way. That feels like the next thing to do. Also for the previous cycle, I had tried to do a position based trigger. So I dissected the circular space of the rug in 4 quadrants, and depending on where a user is, it triggers some visuals on the screens. I couldnt get it to work but I keep thinking what if about it. I would also love to explore some physical interactions triggering some events in a cool physical-digital way.

What I learned across all three cycles: I started off by trying to make an AI listen to the user and ended up making a space that receives the user instead. These are two very different orientations, but I learned throughout the process that making a good experience requires you to pay attention to even the most seemingly-insignificant interactions and feedbacks of the users. Things that don’t even feel like findings when they’re happening are sometimes the most useful data you can collect. It’s just very easy to miss them because we (atleast me – I dont speak for everyone) are looking at the system instead of looking at the people.


Cycle 2: مہمان – Mehmaan (The Guest)

After the unsuccessful debut of my Cycle 1 project, where I had built a conversational system and dressed it in fancy spatial experience clothes. I realized I was trying to do too much and jumping in too quick into the technicalities of what was at the time my very broad area of research with, without having alot of knowledge of the area. Just for reference: my area of research interest is designing an AI companionship experience which embodies the aspects of companionship that a text-based companion would not – especially for South Asian adults experiencing loneliness. So, for the next cycles, I thought it would be great if I strip my goals down to creating an experience that achieves part of it. So, I pivoted to creating an embodied experience where the space acknowledges you, with a flair of South Asian culture. In short: I just needed a body in a space and something the space could do in response.

So, the concept became Mehmaan. Mehmaan means guest in Urdu. Mehman-nawazi (the hospitality, care, and honoring of the guest) is a specific South Asian, or in my case, the Pakistani cultural practice. I translated that into a spatial experience where the host (the space) does all the work. The guests (the users) don’t have to operate or figure out anything. They are just received and honored. That is the opposite of every interactive experience I have ever used, but it felt like the right thing to create after Cycle 1 where the user was expected to do all the labor of making the system work.

Resources

  • TouchDesigner
  • Orbecc depth camera for body/blob detection
  • My trusty laptop (I have not named her yet)
  • The song “Mehmaan” -> https://youtu.be/mtIjUH4aQSA?si=mkKMoMGcJ28JOHrX
  • Top down projector, and THE rug
  • Projection screen
  • Motion Lab
  • Tripod mounted Camera

The score

The user steps into a dimly projected floor of Pakistani geometric patterns (truck art). As the user enters, the space becomes more alive: the projections get brighter, and music begins playing. On the large projection screen in front of the user, flower petals fall and respond to the movements of the user. So, not deliberately doing anything – the space just blooms because someone is in it.

Process

Blob detection from the Orbecc signaled the start of the experience: blob detected -> song starts, floor pattern brightens. No blob, no experience. The space is dormant until you enter it. The projector screen shows slowly falling petals in the background throughout the experience. The floor pattern is a sequence of Pakistani truck art motifs animated with beat detection from the song. I used a mirror top so that the pattern dances with the music.

The optical flow for the petal- particle effect interaction was the most fun part to build. You move, and the petals physically respond to your body. It’s immediately perceptible without any detailed instructions.

What Worked, What Didn’t, What I Learned

What worked: Almost everything that I hoped to, which was great! The feedback told me the piece was very fun and made people want to dance, which was not the plan but it felt right. The cultural aspects like the patterns and the song was immediately warm rather than alienating even to people unfamiliar with it. The optical flow interaction was intuitive enough that people discovered it themselves, which is exactly what I was going for after Cycle 1 where nothing was intuitive. The experience also became very social. It started with everyone taking turns to it turning into a dance-playground for everyone!

What I learned: the performative discomfort of being alone in front of a system watching you is a design problem, not a personality problem. The system needed to give people something of themselves back. They needed to see that they were inside it, not just in front of it.


Pressure Project 3: Love Letters from Home

This project was created as a purely audio experience, a three-minute piece with no visuals. It was meant to be cultural storytelling, but in the process of doing that, it became something very personal. Honestly, I’m not even sure if I made it for anyone other than myself.

The interaction is built around proximity form the camera and the system. Using Mediapipe inside TouchDesigner, the system estimates the participant’s distance from the camera based on the space between their shoulders. That distance is then mapped into three separate zones: near, mid, and far. Each zone triggers a different audio sequence of a minute long, creating a shifting soundscape as you move through space. The interaction is very simple but it gets complex in what it means.

The zones are not just meant to be spatial, I also intended them to be emotional. The farthest zone holds the ambient sounds from back home. It contains sounds of a place I have very fond memories of, Liberty Market in Lahore. It’s full of life, voices, interesting characters, and movement. It holds the ambient chaos and the unique life of a place that feels familiar when you’ve lived inside it. Underneath it also runs the sound of a dhol, a traditional drum from South Asia which is often played during celebrations and festivals. The recording is from my wedding, which turns it into something both public but also secretly personal.

The middle zone moves closer to the system and my life, into family. My parents asking if I’m okay, telling me to take care of myself, giving blessings in the everyday way they do. There are also scattered pieces of time spent with my siblings, just being together, being absurd and being just us. There is also snippets of my dad telling us a ghost story around a bonfire. All of them are just small moments when they’re happening, but they kind of accumulate weight as time passes, especially now that being together like that is rare.

The closest zone is the most intimate. It has audio of me and my husband. Snippets of our vows, pieces of our wedding song, voice notes he sent me from long distance of him singing to me, and also us singing together. The songs are in Urdu, which is our mother tongue. So music in this case, is not just background. It’s part of how we stayed close across distance before we could be in the same room.

The piece doesn’t guide you or explain itself. You move, and it responds. What you hear depends entirely on how close you choose to stand, how long you stay, whether you move toward something or away from it. That felt like the right way to design this piece because that’s how memories also work: existing in a very non-announcing sort of a way.

I asked someone else if they would like to perform the piece (I didn’t feel like putting myself out there and I thought I would end up crying), and Chad volunteered. That did not go exactly as planned and it annoyed me because he was trying to figure out how the system worked and all of its interactions, and in doing so, he missed half of the experience. I realized I shouldn’t have told someone else to give a performance for something as personal as this project in my stead, because it wasn’t a puzzle meant to be figured out.

So I did what I had to, in order to fix the situation: I performed it myself. I may or may not have cried while creating this, but performing this made me so happy and I felt relieved to have done it “properly”.

The most surprising part was that even people who didn’t understand the language still felt something. Lou told me they got teary-eyed. That meant a lot to me. It showed that the piece wasn’t just about language or culture in a literal sense, but about something deeper, and also obviously about the performance since they did mention that they could tell how much this meant to me while seeing me perform.

If I develop this further, I want the zones to be less blocky, and to be more fluid and abstract, even floating around like memories. But even as it is, I really like this piece because of how much it means to me, and it does feel complete in a different way.


AI Expert Audit

I used several publications from Dr. Samantha Krening. I have known Dr. Krening for years and have been really interested in her research and background.  

I was honestly shocked by how good this AI generated content was. It truly understood the Reinforcement Learning with Human Feedback. I could easily hear Dr. Krening saying these statements.  The experience was almost unsettling, especially when we got into the numbers expressed in the infographic. I don’t know these numbers. I can’t validate them or invalidate them. The fact that everything else that it put out made me nervous about whether I would even question them. I’m also concerned that looking at them will imprint them in memory and I’ll forget the source. This is why I try to shy away from AI unless it’s for advice for tasks that I can immediately put into practice and get feedback on the efficacy. If it doesn’t work, I can let it go. If it does, I feel less concern about polluting my own mental space. 

The podcast is fine for a while, but eventually the speech patterns feel manipulative and I grow increasingly uncomfortable listening to it. Eventually it gets to be too much. When the AI is talking to itself and acting like things are amazing, surprising, whatever… when it’s expressing emotion it feels uncanny. I hate it. I suddenly become very aware of the smart podcast host affect it’s putting on.  I stop listening because I get this feeling of existential dread. Sometimes it gets really excited about information that doesn’t seem natural. Stuff that a human isn’t going to get that excited about. It can make you feel like you missed something until you listen again and realize oh okay this thing is kind of empty behind the… lenses? 

If you can’t test it don’t use it. Don’t count on yourself to validate everything the AI says because you’re not going to. It’s going to free up time and that time is going to get spent up somewhere else, you’re not going to reallocate it to validating the AI output over the long run (this is the law of stretched systems). Also be hyper vigilant about what you’re taking in. The fidelity of AI generated output is remarkable today. One day in the very near future it will be indistinguishable from non-AI generated content. Even if you’re not intentionally using AI you’re going to be ingesting AI content. Eventually that content is going to be hyper-optimized for micro-targeted persuasion. If the AI can be this good based on 8 documents, imagine what it will be able to do when everything you have ever written on an electronic outlet is purchased and used to train a machine to convince you what to buy, what to think, who to vote for. Imagine how effectively it will be able to report on your most probable next response, behavior, or obsession. We are using AI today, and we need to understand that AI is much better equipped to use us.