interaction design • tangible user interfaces • Arduino • Processing
In painting, novices are often too quickly discouraged for not being 'good' at art. Painting with a partner can be valuable to spur creative inspiration or for overcoming mistakes. The benefits are similar to those of pair programming or dancing with a partner. Although, in painting, these benefits are difficult to achieve because painters are fearful of ruining the other partner's creative vision.
Empainty (a mechanical watercolor palette) is a tool that shifts the focus of painting away from a single vision for the piece, towards exploring the painting process itself (specifically colors and brush strokes). Empainty uses gamification to promote collaboration, creative inspiration, and problem solving between the partners, alleviating fears of ruining partner's work. Here's how Empainty works:
One Empainty color palette. One canvas. Two painters.
Partner A has long fluid strokes. Partner B has short splotchy marks. Since they are using different strokes, Empainty limits them to cool colors of green, blue, and purple representing the partners being cold towards each other.
As the partners start using similar brush strokes they unlock warmer colors. You can see they've unlocked yellow on Empainty.
Once the partners are fully synchronous, they unlock the warmest colors of red, orange, and yellow representing their warmth towards one another.
(Sketches by me)
Users successfully shifted their focus from a grand vision for the painting, to the process itself.
"We were focused on changing the color boxes rather than focusing on the painting."
“I think it's encouraging me to try different patterns and different textures.”
Project team members: Dina Bseiso • Jake Petterson
I led my team through multiple design iterations pushing us to brainstorm broadly and allow our users to heavily influence the design. The following highlights some of our process.
From early design iterations, we were interested in making the artistic process more accessible by exploring how people share this process and affect one another’s creative experiences. We noticed drawing was an individual exercise and questioned how it could be more like dance or music where one feeds off the energy of a partner.
Our initial concept to explore this was Emotive Ink. Conceptualized as a product shared between partners who were in remote locations, each partner with their own ink pen and ink bottle. As one partner used the pen, the ink would change colors reflecting the other partner’s emotional state. And, vice versa for the other partner.
The following scene shows the partners working in different locations from each other. The first partner (top) is angry (expressed in red), so the second partner (bottom) receives that red ink color to work with. Vice versa, the second partner (bottom) is calm (expressed in blue), so the first partner (top) receives that emotive color of blue.
The following are sketches of the ink bottle and pen. (Top) The ink bottle sensing when the ink pen is being dipped into the bottle to digitally ‘pick up’ the emotive color of the other partner; (middle) the ink pen mechanics showing the release of colors from cartridges and the colors mixing to create the emotive color of their partner; (bottom) the ink pen biosensors (ex: sweat conductivity, finger pressure, etc.) capturing the emotional state of the partner.
(Sketches by me)
Empainty v1 Design
Given the small space of the Emotive Ink pen, the technical implementation was ambitious for our deadline. So, we iterated and switched mediums to painting, with Empainty. Painting allowed us to limit colors available to the user while giving us room to house electronics in the color palette.
With Empainty, instead of capturing emotions we focused on capturing the energy of the pair (measured by how similar and dissimilar their brush strokes were to each other). Measuring of the brush strokes also served as a way to focus on technique, as opposed to outcome, something inherent to the artistic process.
To map similarity of brush strokes to colors, we used ‘warm’ colors to represent closeness and ‘cool’ colors to capture distance. Restricting and allowing certain colors based on how the partners are working together allows for a reflection of the warmth of their relationship both in the resulting piece on the canvas and in the patina on the wooden palette box (wood chosen in fabrication for this purpose).
The lids over the paint wells on the palette box have a fluttering effect to bring a sense of life and personality to the inanimate object. For example, if the partners are close to similar brush strokes, the box will flutter open to the ‘warmest’ state and then close, encouraging the partners to continue working together to unlock the warmest color. Instilling a personality into the palette box allows it to be a third actor in the collaborative painting experience, both encouraging the painters and remaining as a neutral party in facilitating the collaboration.
The following are sketches of the watercolor palette box. (Top) Fluttering lids opening and closing over paint wells; (middle) inside view of the palette box showing servos housed underneath the paint wells operating the opening and closing mechanic of the lids; (bottom) backside of the palette box showing the living hinge mechanism.
(Sketches by me)
The living hinge mechanism ultimately failed, so we adjusted the design such that a servo arm would attach directly to each lid. This gave us more reliable control of the lid motion. The following picture shows this failed living hinge mechanism and how the design changed through the prototyping process (see more details in technical implementation section).
Based on observations of users interacting with Empainty, future versions will focus on (a) personalizing the brush stroke gesture recognition and (b) making it more evident to the user how the palette box responds.
In the current version, painters are locked into pre-trained gestures which are not specific to them. Future versions would have gestures trained as the tool was used, personalizing them for each user. Additionally, more sensors could be used (flex sensor or FSR) to better differentiate brush strokes.
Some users did not immediately understand that the palette box responds to brush stroke similarity. To make this more evident, the palette box would respond to the extremes of brush stroke similarity and then over time become more difficult by responding to more nuanced differences. In this way, users can understand the concept on the simple case and then gradually be challenged.
See the full report for more discussion on the design and future iterations.
I supported the user experience research by conducting observational studies and developing interview and survey questions.
We conducted two observational studies complimented with interviews and surveys. One study involved two painters collaboratively painting on one canvas without Empainty, and the second study conducted with two painters collaboratively painting with Empainty.
Collaborative Painting Observational Study
In this study, we had two painters collaboratively painting together on one canvas. The two painters shared a single color palette and were simply instructed to paint. Empainty was not used in this study. In the second half of the study we manually restricted certain colors (similar to how Empainty does) to see how the painters reacted. After completing their painting, we asked the partners how they felt about the experience and challenges encountered.
The painters began on sections of the canvas nearest to them almost creating an imaginary boundary between the two painters. They were apprehensive to paint in the others' space in fear of ruining the piece, and felt limited in the collaborative environment:
"Am I ruining your thing?...I think I messed it up."
"I don't want to mess up anything, because it is a collaborative effort...maybe I would keep going [if it was just me]."
Despite fear of ruining their partner's work, participants did gain creative inspiration from their partner, supporting a benefit of collaborative painting:
"Oh, what'd you just do, that was cool!""
When the constraints on the colors were imposed, some immediate reactions from participants were of surprise and confusion. In all cases, a dialogue occurred between the two painters - something that was not evident in all pairs prior to imposing the constraints.
Empainty Pilot Study
With version 1 of Empainty, we conducted a pilot study with users. First, we interviewed each pair of participants about their views on creativity. Then, we observed the pair of painters interacting with Empainty. Afterwards, the participants were given a post-activity survey. When using Empainty, the pair was simply instructed to paint, and the palette box immediately responded to their gestures. We did not disclose how the palette box responds to their brush strokes, wanting to see how their behaviors might transform as the interaction continued.
In many cases, the participants' focus shifted away from the piece, to the process itself:
“I stopped caring about making a good painting.”
“I did not try to represent real objects. Our art evolved more organically.”
“Till the end, I didn’t know what I wanted to paint. I was just exploring different colors and strokes and having fun.”
Users also found that Empainty encouraged them to be more creative. In response to "Describe a moment, if any, that pushed your sense of creativity":
“Discovering a way to paint what we wanted while maintaining the motions necessary to get the colors we wanted [pushed my sense of creativity].”
“I saw my partner was working abstractly so I tried to fit into that aesthetic more (P01). I saw my partner make dots and thought ‘I can do that’ (p02).”
We were able to look at the resulting paintings from the two studies (with and without Empainty) as a way to see how Empainty changed the focus of the activity. The following paintings created without Empainty appear to have a focus on a specific image or scene:
In contrast, the following paintings created with Empainty are liberated from a specific vision and instead explore a wide range of different brush strokes. Evidence of Empainty shifting the artists' focus from the end result to the process itself:.
For a more detailed discussion of the studies conducted, findings, and future testing, see the full report.
I led the technical implementation by designing Arduino circuitry with servos and accelerometers; pair-programming Arduino and Processing code and the supporting C libraries; designing TCP communication architecture to send data between the accelerometers and servos; and training the brush stroke gesture recognition algorithms.
Full code and implementation instructions are available on my github.
Prototyping the Palette Box
(Box construction by all team members)
As seen above, we started with a cardboard prototype with a living hinge mechanism for the flapping lids (upper right). We then moved onto a laser cut version (bottom), where the living hinge mechanism failed (center). In the final iteration, we elected to have a more reliable mechanism by attaching the lids directly to the servos (upper left). Here are the box dimensions for the laser cutting:
(Illustrator for box dimensions by Dina Bseiso)
Each of the paint brushes had an accelerometer, and the palette box used servo motors to move the flapping lids over the paint wells. Here is a schematic of the circuitry and the actual implementation.
(Circuit design by me)
Brush Stroke Gesture Recognition
On each of the paint brushes we placed an accelerometer on the tip of the handle. We used gesture recognition software to train brush strokes on the x, y, and z acceleration data.
Here's an example of the data training examples for 4 different brush strokes. A 'poke' stroke is on the far left and a 'big-swipe' stroke is on the far right.
(Gesture training by me)
Can't forget the late nights debugging with pseudo-code on messy whiteboards (I'm on the right) -