Following the proposal presented in this link, IMERSA became pretty much what I had anticipated. In this post I will expose the challenges in this process, and the decisions taken because of them. I am not used to thinking the creative process before doing it, and this was also my first long creative process with no collaboration, which changes a lot in the dynamics of the work.
[I am not yet in possession of the documentation done in the winter show, so I’ll upload this post with it once I have them]
The first part of the process was figuring out the fabrication of the piece. For both environmental and budget reasons, I was focusing on recycled materials. The wood decision was made based on the materials that I could find. All my prototypes were made with scrap wood from the floor, and I got some scrap wood from the theater department, so that was the material used in the final piece.
I was planning on making it 60x60x60cm, but the size of the prototype worked just fine, so I decided to make it smaller. I calculated the size of the hexagon in the wrong size of it, so it ended up being even smaller than I expected, which worked even better than the planned size. The acrylic for the bottom of the cone was left over of a friend’s experiment, so it had a few etchings that were not planned, but looked interesting in the final piece.
I created two hexagonal wooden prisms, one inside if the other, to support the top and the bottom of the mirrored cone. The inside prism had to have enough space between its pillars to accommodate a projector, the bigger one had to have space for the motors, and the distance between them should be able to accommodate a speaker, the arduino, and the breadboard. I tried screwing the structure, but the wood pieces were very thin, and the screws available in the shop were too big for them, so I ended up breaking the prototype wood, and decided to glue most of the final structure. Also, the angles were pretty tricky, because I’d need to screw the joints of the hexagons to the pillars.
I used glass crystals as the shiny pieces inside, and I really enjoyed the process of creating the shape for it. The metallic rings are certainly not the most sturdy material, so I feel like they are the most fragile part of the final piece, and I’ll certainly have to change them a few times if I continue to present the work. I was afraid the glass would chip a lot when it fell, but it happened way more when I was creating if then if the actual presentation, so I believe it may hold for a long time.
In the first prototypes I used iridescent film to make the hexagonal cone, and I figured that it could be used in the final piece since it was cheaper than the acrylic mirror that I wanted to use, had a nice colorful effect with the materials inside, and the webcam would work through it, hiding it better than making a hole in the mirror. But, the stage design department was getting rid of an acrylic mirror in the perfect size for my piece, so I went back into the idea of the mirror, but kept the size of the iridescent film and used it over and over in my experiments for it’s flexibility. I had some trouble with the laser cutter, which made two of the cuts to be imprecise, and even after sanding they didn’t fit as well as the other ones. The attachment was tricky anyways, so I decided to tape them together. I forgot to lasercut the hole for the webcam, so I ended up doing it with the drill. I tested it in another piece of mirror and it worked fine, but when I did it, the mirror chipped around the cut. I opened the hole a bit to try making it even, which helped, but it was still a bit chipped.
The pulley system was probably the trickiest part of fabrication. I had bought sewing machine bobbins to use as spools, but in the end the shaft hub did the work by itself in the bottom, and at the top, there was no good way to attach it. I made a few holes in the acrylic to let the string through, but it would get stuck, so I needed something to actually hold the thread once it was out of the holes. The solution was to attach hooks as parallel to the structure as possible, above the holes. The thread ran from the crystal to the hooks, passing through the hole without applying any pressure to it, and then back to the motors in the bottom of the wooden structure.
The motors were mounted as near the corners as possible, and since the screws available were a bit longer than necessary, I attached three feet to the structure that ended up being essential for the cords to run out of the structure, and for the projector to have space for ventilation. I had some trouble with the string I bought, because it was stretchy and rubbery, so it got caught very easily and stretched whenever that happened. Luckily a friend had a very thin non-stretch string that worked perfectly in the system.
The final part of the fabrication was the outside. I was honestly sad to cover such a nice structure, so I decided to make it translucent. It doesn’t really show the structure, but it is sure nicer than it would be if it was the black opaque box that I had planned initially. First I bought fabric, and realized that it wouldn’t work the moment I put it near the structure. Then a friend suggested paper, which looked really nice, but I bought a paper that was too thick for folding, so it would break whenever I folded it to fit the structure. The final material was also something that I got from the stage design department. I don’t know the name of it – I’ll edit when I get that information –, but it is a paper-like very sturdy and flexible material. It seems like there’s some amount of plastic in it when you try ripping it. It’s texture is very nice, and gave the piece an organic look that I really like.
The last part was the mirrored lid, to add some nice finishing to the top. Again, I had some trouble with the laser cutter, and it ended up cutting the pieces in the wrong shape. I tried redoing it, but some of the material was already compromised. The acrylic glue also didn’t work the best with the coating behind the mirror, so it didn’t glue very well. In the final piece I had to tape the hexagon together. I plan on redoing this part for future presentations.
Final sketch link: https://editor.p5js.org/mardefronteira/sketches/HyARG-AyN
I wanted to use gaze tracking to recognize if anyone was actually looking at the sculpture, but the resources that I found for that were either not working or too complicated for me to implement in the time that I had, so I decided to use PoseNet to recognize if there were any humans present, but that library would cause it to trigger not to anyone looking at it, but to anyone around it, so I had to put the sculpture high enough so that people would need to actively look into it, sometimes even needing a step stool, depending on the person’s height.
PoseNet creates an array of people, each containing an array of tracking points in their bodies. With that, I made a very simple conditional to tell me if there was a someone in the frame and, if so, if there were more than one person. One person would cause the program to say “play”, more than one would cause it to say “pause”, and no one would cause it to say “reset”. These were later on transformed into the status to define which step of the sculpture to play.
The sketch is divided in three main classes: Shape, Sound, and State.
Shape is the lighting, being defined by a timed animation consisting of three scenes separated in the functions walkers(), spiral() and lines(). Both walkers() and spiral() use a Particle class as units to create the shapes. The idea was that the particles would move softly from one shape to the other, but the program would run very slowly after a certain number of particles were created. The solution I found was not using the particles in lines(). After playtesting the projections, it was clear that it was too small for the piece, so I modified the circles that compose the lines to grow in diameter and change in color over time, eventually filling up the screen.
The sound is composed by two sets of voice recordings that I did, one with spoken text, and the other one with singing. These recordings are layered up in different ways in each state, receiving volume and speed modifications. Just as the light, the sound increases in mass, filling the space with sound over time.
Both sound and light are triggered by the State class, which relates to the data sent to the Arduino to trigger the motors. The three states – play(), pause() and reset() – are defined by PoseNet, as explained earlier. While in play(), the projections start and follow its timeline, going from walkers() to spiral() to lines(). The sound starts with one singed note, in lower volume, under a louder text recording, and other note layers are added. The texts are usually just one or two simultaneously, depending on their duration. The crystals are brought up by the motor in steps – I’ll talk more about that in the next section. If the state changes to pause(), the shape becomes static, the sound lowers its volume, even though it continues to increase in density, and the crystals stop going up. If it changes to reset(), the crystals drop abruptly, light and sound timelines are reset, all shapes disappear, and the sound becomes a mass of spoken text in a very low volume.
The serial communication is pretty simple. p5 sends either the number 1,2 or 3 to the Arduino, corresponding to the states in play. The Arduino reads this value and applies it to its own status, which I’ll explain in depth in the next section.
In the version of the piece that I presented for Intro to physical computing, the webcam was positioned on top of the structure, so it would see people that were not looking into the structure. But, when I made the hole for the webcam, the angle narrowed a lot, so the pause() function doesn’t work as precisely as it did before.
This was by far the most challenging part of the work. The first decision I had to make was considering which motor to use. My teacher lent me a gearbox DC motor for me to try the first prototype, and it did lift what I needed it to lift, but it was too fast and not strong enough to bring the pieces up, and too slow to bring the pieces down.
I then had office hours with a resident who told me that the motor was enough, but if I wanted more torque I could try an interchangeable gearbox. The thing is that the speed of the falling crystals was the most important thing for the concept, so I decided to buy a motor that was both stronger and faster, but at that point I still had no idea how the speed X torque trade worked, therefore I ended up buying a very fast motor that wasn’t strong enough.
I got the motors to go up, down, and stopping quite easily. One of the residents gave me the idea of using limit switches to tell the Arduino when the crystals had reached the top and the bottom, so I did the code for them too. The problem is that when I tried making the motor slower, it would be very weak, and sometimes even wouldn’t be able to spin at all. Also, there was no way to install the switches, because there was only tension in the string when it was pulled up, so there was no way to put enough pressure on the switch when coming down.
I figured that timing the motors was the only viable way of doing it, so I started testing with delays, since they are easier to deal with, so it would work, but it would take very long after the state changed in p5, because it would only read again after the delay was done. I tried doing it with millis(), but for some reason it just wouldn’t work. The code was fine, but it was going a bit crazy and working very differently from the delay.
After more office hours with residents, I was able to develop a step system for both bringing the crystals up and down, and setting more points of verification for states during the movement. With that, I was also able to establish the limits through math, not needing the limit switches anymore.
In the end, it worked like this: starting the Arduino would pull all strings to the limit, calibrating the point of maximum steps, and shortly after it would fall for a certain amount of seconds, calibrating the point of zero steps. When the status changed to play(), the motor would pull the crystal up for a fraction of second, and then check again to see if anything changed. If not, it would make a new step, until it got to the top. At the top, it would be pulled a bit more abruptly, so that it also recalibrates the distances on its own. If the state changed to pause(), the motor would stay in place until there were any changes. If state changed to reset(), the crystals would fall the amount of steps that they are up times the time it takes in a fraction for it to go down the same amount the it went up in each step. This is what the whole cycle looked like:
This project was presented in the ITP Winter Show 2018, and since it is all about giving the piece time and patience to be there with it, I didn’t want to be near the work explaining what to do and observing people. I tried staying away for a while, but most people would just look a the outside and not even try anything with it. I realized that the show wasn’t the best form of display, so I decided to stay around and give some instructions to people.
In the second day, I left written instructions, and I feel like it worked just fine. I still want to find a way to do it that looks more cohesive with the piece, but the words themselves seem to be enough. I wrote “look inside” near the top edge, and “use me if you need” on the step stool.
The height was a problem because it was decided so that very tall people could use it without the stool, and shorter people could see it with it. But, sometimes very tall people would use the stool, putting themselves too high for the camera. Also, as I mentioned before, sometimes the camera wouldn’t identify more than one person, so it wouldn’t pause when it was supposed too. Also, because I was located in the dark room, I had to put some extra light for the camera to see people.
There are still many things I’d like to change going forward with this project, such as making the sound more understandable, because I am dealing with vulnerability, so I feel like I should be more exposed. Another point would be making the projections more relevant to the interaction, so that the feeling of growth would be intensified, and soothing the movement of the motors a bit, because some times I thought people would fall out of the stool because of some intense sudden moves.
I am very happy with what I was able to do with the time frame I had, and I really wish to go forward with this someday. A lot of people mentioned they would love to see it in a bigger scale – which was the first idea – so maybe one day I can get that done.
Lots of gratitude to Luisa Pereira, Jeff Feddersen, Yeseul Song, Mithru Vigneshwara, Verónica Alfaro, Aaron Moreno, Dana Elkis, Anna Gudnason, Adi Dahiya, Arnab Chakravarty, Ellie Bye, AJ, the dark room people in the winter show, and all my classmates in ICM and pcomp. Thank you so much for your generosity and feedback on this project and/or for the shoulders and ice cream breaks.