June

Fearless

Nadieh
amsterdam
Cardcaptor Sakura | Fifty chapters of adorable cuteness

week 1 | data

Yes, I know it's been 6 months since I finished up the April month, and yes, I know I'm supposed to do May before June. However, I'm still not quite sure about my angle for May and I did have an idea in mind for June πŸ˜…

I am starting to understand that the topics that I can get really enthusiastic about are a bit of a niche. That not many other people truly know it. I can only hope that for the people who are fans, as I am, this month will be a joy to explore. The topic that I've chosen for our month of fearless is "Cardcaptor Sakura"!

Cardcaptor Sakura

A magic-girl manga (i.e. Japanese comic) from about 20 years ago. It was the first manga I owned, when manga was practically unknown in the Netherlands (still is actually πŸ˜• ). I even had to travel all the way to the biggest city of our province to buy a new volume. I'm still dreadfully jealous of how perfect and cute practically each panel of the manga looks 😍 I almost chose this subject for our January Nostalgia month, but eventually went with Dragonball Z then. Nevertheless, after 20 years a new "arc" of Cardcaptor Sakura has started again recently. Therefore, while thinking of what to do for our fearless month, I just couldn't shake the feeling of wanting to do something with Cardcaptor Sakura (CCS), a brave, fearless, and kind magical girl.

One of my favorite things about CCS is how beautiful the writers, CLAMP, make each page. Especially the covers of each chapter, which are tiny works of art (the image above is the cover for chapter 23). I therefore wanted to investigate the covers through data somehow. And I've never before done any kind of analysis based on image data. I therefore thought that creating a visualization that would abstract the colors of each cover into 3 - 8 colors would be fun and new for me.

There are 50 chapters in the original CCS manga, divided into two "arcs". I went through all 12 CCS volumes to see what image was on the cover of each chapter (btw, you can also read CCS online, for example here). In my volumes the covers are printed in black and white. However, all of those chapter covers have since been published in full-color in several CCS art books. I therefore searched for and downloaded the corresponding color image from the CCS Wiki page.

All 12 manga volumes of Card Captor Sakura

Using the imager package in R, I loaded the images into R where each pixel was transformed into a multidimensional array of RGBA values. I converted that complex array into a more simple data frame of (number of pixels) * 3 (for r, g, and b value) size. To figure out which algorithm would cluster the pixel values into decent colors groups I tried several things. First I experimented with using different clustering techniques: from the standard K-means, to hierarchical clustering and even tSNE. But I also converted the RGB values of each pixel into other color spaces (where colors have different "distances" to each other and can thus result into different clustering results), using, amongst others, the colorspace package.

I often converted the results of each test into a bar chart such as below to see the color groups found. Eventually, I found that using Kmeans together with the colors converted to "Lab" visually gave the best fitting results.

Color distribution of the first CCS chapter using Kmeans

However, one of the tricky things with K-means is to figure out how many clusters should be used to create groups. I first tried a combination with hierarchical clustering, but eventually I decided to use something that was probably a better judge (but more time consuming), my own eyes! For each chapter I created a graph as below, that shows me the color distribution for 3 color clusters, up to 11 (I didn't want too many). I then compared the actual cover to these groups and chose the best fitting one; a balance between capturing all the colors and having a good blend of distinct colors. I saved the hex colors and %s (the height of the bars) of the best clustering into a json.

Color distributions of several Kmeans results with different number of clusters

To complement this data about the chapter covers, I also wanted to gather information about which characters appeared in each chapter and which "card" was captured in which chapter (CCS is about Sakura collecting so-called magical Clow cards). The CCS Wiki page on each chapter seemed like just the resource I needed. But sadly, only the first 8 chapters contain information. Well, nothing else to do but to read all chapters again myself while slowly filling an Excel file with the info I needed 😜

Due to the "layered" visual result of this month, I eventually sliced and diced all of this information (who is in each chapter/cover image, totals per character/chapter, relations between characters, color distributions, and more) into about 7 separate small files throughout the creation process. I prefer to prepare all data beforehand in R. I find that the easiest and as a bonus, it doesn't clutter my JavaScript code.

week 2 | sketch

Figuring out the design for this month came slowly. It was more a domino effect. A more concrete idea for one aspect led to a vague idea for the next part of the data which I then explored. I started with how to visualize the colors. Having a cluster of small colored circles per chapter seemed like a logical/interesting step. Placing the color clusters in a radial layout was also an obvious choice after that. Although at first I wanted to do a semi circle, with the color clusters to the right and character info the the left. However, with 50 chapters, I really needed as much space as I could get. So that's why I went with the layered approach of a small circle for all characters and around that another circle with all chapter color clusters. These two circles would then be connected by lines to show which characters appeared in which chapters.

Sketch of the overall design for the CCS visualization

I've always been fascinated by the CMYK dot printing process; where you can see the separate dots when you're looking at it up close, but move farther back and the bigger picture comes into view (I guessing I'm not the only one who could sometimes be found with her nose literally touching an old magazine, or old TV (for the RGB stripes), right....?). Recreating this CMYK dot technique for a visualization about a (printed) manga seemed like a proper style, and challenge. And challenge it was! I won't go into the details here (you can read a bit more in the code section), but below on the right page you can see some scribbles I made to understand how to recreate the CMYK effect (it has to do with rotations...).

Sketch of trigonometric functions to figure out CMYK dots

Another mathematical challenge this month was for something that I didn't use in the end... At first I wanted to connect the inner circle of characters with the outer ring of chapters with swirling lines (to say it technically, two connected SVG Cubic BΓ©zier Curves). Making sure that these lines would always flow around the inner circle and look good, took a lot of time and notes in my sketchbook! I always prefer to draw the approximate SVG path shape I have in mind to then try and figure out where new points and anchor points should be placed. The really hard part is to then understand how these points and anchor points change when the data changes; how to create a "formula" that works for all instances.

Sketch of the swirling lines between characters and chapters

The page below shows specifically how to handle the calculation of a tangent line to a circle for different circumstances. I needed this information for those swirly lines from above. But even though I ended up with different lines, I could thankfully use part of the things I'd figured out on these two pages to easily convert the lines to what became the final result (the more circular running lines).

Sketch of figuring out tangent lines

week 3 & 4 | code

I first focused on getting the ring of cover color circles on my screen. Mostly because I wanted to see how that CMYK idea would look as soon as possible. And thanks to this excellent example of multiply points of gravity by Shirley, that was actually a piece of cake! But damn, those circles had to become quite small to make room for all 50 chapters. I was starting to get my doubts if the CMYK effect would work as well in this particular design as I'd hoped...

All the main colors of the 50 chapters clustered

Nevertheless, I first went to one of Veltman's amazing blocks in which he already neatly coded up a CMYK dot effect as SVG patterns (btw, I've started collecting my favorite d3.js blocks in a Pinterest board, so I have a visual 'bookmark' for each, which makes for easier retrieval). Rewrote that to create a separate pattern for each color and I had myself a ring full of CMYK based colored circles. But on closer inspection I found something I didn't quite like. Although the circles on the inside looked exactly how I wanted them, they were still SVGs. So they had been perfectly clipped into a circle. Truly like a pattern that you cut off. But I wanted my CMYK dots to smoothly fade out, not abruptly. But I also wanted to play with the idea of partially overlapping the circles, and having the colors mix even further, which wasn't possible with this technique.

A selection of CMYK circles

I therefore did a wide search online looking for other examples. I already expected that using HTML5 Canvas was probably the way to go. And I did find two interesting options here and here that took me a good 3 - 4 hours to wrap my head around and combine into one. It took a lot of testing an tweaking...

Testing my code to create canvas based CMYK effect

But eventually I got the visual options and look I was going for. First, smooth edges. By which I mean that the dots get smaller around the sides, but all the CMYK dots are still full dots. But also that I could plot the circles on top of each other with the CMYK effects of both circles visible.

Final canvas CMYK circles than have smooth edges and can overlap

And then I applied it to all the circles from the 50 chapters and..... as expected the circles were so small that there wasn't "enough CMYK" going on. It was sometimes a bit hard to actually get a feeling for the true color of circle because it contained only a few CMYK dots in itself πŸ˜–

Well, that's how (dataviz) design sometimes works, hours of work on something that never makes it to the end. So I converted a simple version into a block to perhaps use for another time and took a closer look at the original SVG version again. What to do about those crisp outer edges? ... Hmmm ... What about adding a thick stroke? And yup, that fixed it enough for me, haha πŸ˜…

I thought the lines between the inner and outer circles would probably be the next most difficult thing to tackle, but to properly do that I first needed my inner circle. Whipping up the thin donut chart from my sketch was straightforward with d3's arc and pie layout. Still, I felt I first wanted to see if I could get the connections/relations between the characters "visually working" on the inside before I moved to the outer lines. Because if it didn't work I might have to think of a different general layout.

Time to write some custom SVG paths again! Below you can see the progress from the simplest approach, straight lines in the top left, to the final version (shape wise), in the bottom right. The final version is made up of circles, using the SVG arc command, re-using code that I had written for the small arcs in my November piece about Fantasy books.

Figuring out the SVG paths of the inner relationship lines

I then colored the lines according to the type of connection (e.g. family, love) which made me see that there weren't too many lines in there to get insights from, no visual overload, pfew. Alright, then it was really time to dive into those outer lines...

To most extreme lines that I could create would run from a character to a chapter that's on the other side of the circle. The line would then have to swirl around the inner circle, without touching any of the other character's names. I thought I could probably pull that off by combining 2 Cubic Bezier curves. But making 1 of those curves act as you want, depending on the data, can be a hassle. And I found out that 2 was more than twice the hassle o_O

With difficult SVG paths I always start out with placing small circles along the line path itself (the red one in the center below) + the anchor points (the blue, green and yellow-orange one. The pink one I placed for another reason that's too technical for me to explain here, hehe)

Placing the SVG path anchor points to understand the line movement

After some manual tweaking with fixed numbers I had a shape for the longer line that I liked. I saved those settings and did another one, a short line. I then inspected how all of the settings changed between the two options. This gives me hints on how to infer several formulas that will hopefully create nice looking lines, no matter the start and end points. But, like I said, that wasn't as easy this time as I'd hoped...

Understanding how the SVG path anchor points move for a shorter line

Ugh, I don't even want to really think back on what journey eventually led me to have the lines I needed. Most of the notes in my sketches section are about this part. Because slightly different things need to happen when the line moves counterclockwise instead of clockwise. And if you mess it all up completely, well...

Lines going all over the place

Over many, many hours of testing, drawing, thinking and fiddling did I inch closer to having all the lines at least sort of going around the center. Although here the finer details of the lines were still quite.... odd...

Weird looking, but generally correct lines

I didn't make a note of how long this particular section of "creating the lines" took, but my guess is somewhere between 8 - 10 hours. After which I was left with the following when I visualized all of the lines (they represent the chapters that each character appears in):

All the lines, correctly and at once

Awesome, that looked like one big mess! There were too many lines in there to glean any insight. That would mean I'd have to create some sort of hover interaction that only shows you a subset of the lines when you hover over a character or a chapter.

Ugh, enough time spend on those lines for now. I therefore moved on to adding the chapter and volume numbers. The inner donut chart inspired me to try something similar for the chapters as well. A donut chart with 50 equally sized, rounded-off sections, in which I could place a number. I was happy with the end result and could quickly move on.

For the volumes (typically a collection of Β±4 chapters) I started out with the same idea; a donut chart, but made even thinner, which I placed outside the ring of circles. Hmmmm, wasn't as happy with that, but in the meantime something else was bothering even more. Now that I'd placed more elements on the page and all of it seemed to get a sense of "consistency", those inner lines just felt way off πŸ˜•

Perhaps they should have more "body", by making them tapered, as I did in my January visualization about Dragonball Z? That again took more time tweaking my line formulas... Although I think it did improve things over the same-width lines I hade before:

Adding the chapter and volume indicators

And in of itself it had something nice going on when I implemented the hover interactions. Such as seeing who was in a particular chapter...

Tapered central lines for all characters that appear in the hovered chapter

...or when hovering over a character to see the chapters they appeared in...

Tapered central lines for all chapters that a characters appears in

... However, I just felt that it didn't fit the rest of the visual in terms of design. Ugh! What to do about it!? And all those hours of work! 😭

And suddenly, very quickly after knowing that my swirly lines just weren't right, I had a new idea for the lines. I don't even remember what inspired me, it seemed to come out of nowhere (although that's never truly the case). Instead of making them swirl around, I could also make them run along circular paths. A bit like subway lines or piping in a home, but then transformed to some radial layout

This idea was actually quite easy to execute. I could loop over each line to be drawn, create a tiny array of [radius,angle] points and feed that to d3's radialLine function. Together with setting an interpolation function to curve the edges just a bit. Calculating the small array of radii and angles to feed to the d3.radialLine was a walk in the park compared to my previous cubic bezier curve shenanigans! (but actually enjoying to solve geometry puzzles does help). Naturally, all that didn't go right on the first try. And thankfully I could use some of the work I'd done with the swirly lines. The screenshots below were all made within 1 hour, not bad in terms of progress πŸ˜‰

Different steps in the process of creating the 2nd iteration of lines

O, and then I converted all those lines to HTML5 canvas by using the extremely useful .context option that is available in many of d3's drawing functions (such as d3.radialLine). That made things run more smoothly on the hovers!

Final look of the eventual lines between the two circles

With that change of lines, I felt that, when hovering over a character or chapter, the resulting lines fitted perfectly with the "straight-roundedness" of the rest (I hope that made some sense, hehe). And as an added benefit, no more lines were overlapping!

The observant person will notice that I've actually implemented two slightly different line drawing "types". One is drawn by default when you're not hovering, but also when you hover over a character or color circle group. But another is used when you hover over a chapter. Try it for yourself and see what changes πŸ˜‰

The possible interactions in the final visualization

Since part of the visualization was about the covers of the chapters. And because I knew most people that would land on the page would probably not know about CCS, I wanted to incorporate some of its imagery. And I just so happened to have a nice large circular area in the center :) It took a while to manually "cut-out" a good looking square image from all 50 chapter covers. But at that time I was in an airport and on a plane anyway (coming back from a great night at the Information is Beautiful Awards where data sketches won GOLD!! Woohoo!).

Hovering over Sakura reveals an image of her

With all these elements and layers of information in the visualization, I really needed a legend. After having written lines and lines of code to create custom legends in two recent client project (such as this one for Article 19), I wasn't in the mood to do that again. Therefore, I created my legends in Illustrator instead. That saved a lot of time over creating them through code.

Legends explaining how to read the visualization

I initially placed these below the visualization. However (as I was getting used to in this month) these were not the final legends...

But let's not get ahead of myself. Now that the chart itself was nearly finished, I focused on general page layout and annotations. For the layout I had a lot of trouble coming up with something that looked even remotely interesting. To be honest, I'm still not that happy with the final result, but I just can't really design a webpage in itself, just data visualizations, hehe πŸ˜…

But then having to make that layout work on both mobile and desktop.... 😫 Not something I enjoy or want to recount here. Just know it took effort and time.

Early iterations of the general layout and annotations

While reading through all the chapters I took some notes of interesting story points that I wanted to highlight. Using Susie Lu's excellent d3-annotation library adding these around the circle was quite straightfoward. Especially with the super handy editmode, that let me drag the annotations around, see where I wanted them and then add those locations hardcoded back into my code.

But after I had placed all the annotations (see the right visual in the image above) I wasn't happy enough with the result. Typically I love the lines that run from the point that you want to annotate all the way under the lines with text. But here that was getting too much visual weight.

I therefore wanted short lines that would radiate outward from the main circle, and place the annotation around those, typically centered. And although the annotation library gives you a lot of freedom, that particular design isn't in there. So instead, I created my own lines, and then used the editmode (see the small dotted circles below, those you can drag around) to position the annotations exactly where I wanted them 😁

Using the editmode to place annotations

After I felt the visualization was ready enough, I shared it with some friends to ask for feedback and got great suggestions to improve on the interactivity understanding. But one also gave me a great example of a better legend, one that would show the visualization with its rings and explain what each ring truly meant. That was much better than those 3 separate ones that I had before. So I drew a new legend in Illustrator.

And after all that time and effort I finally had a visualization to share with everybody :) which can be found here Cardcaptor Sakura - Fifty chapters of adorable cuteness

Cardcaptor Sakura - Fifty chapters of adorable cuteness

This month took me 86 hours to create. However, a lot of that time went into things that were not used in the final result. Such as Β±5 hours on a CMYK canvas based dot effect, or Β±15 hours on swirly lines and also Β±6 hours on a page layout that isn't even really visible on the final page (except when you press "read more"), or Β±2 hours on stupid Chrome bug about horizontal scrolling (and how to come up with a "fix")

Still, I'm quite happy with how the visualization turned out :) I feel I haven't quite seen a similar radial visualization like it before. And data sketches is all about experimenting with new ideas, so it's always a joy when one turns out good ^_^ I hope you enjoy interacting with the visualization, even if you've never heard of Cardcaptor Sakura before (it's great! Trust me ;) )

Shirley
san francisco

I just want to say how absolutely relieved that I'm now finishing my last month of Data Sketches. For a little while I was feeling a little bit of despair that maybe I'll just never be able to finish this last project. Indeed, I spent two years trying to create a data visualization with Taylor Swift's work - first with the colors in her music videos, and then with the emotions in her lyrics (detected by IBM Watson's Tone Analyzer). I learned a lot from the process, and I really liked the visuals that I had come up with, but I couldn't figure out what I wanted to tell with the story. I was stuck in limbo for a year. (In-progress visualizations of the colors in Taylor Swift music videos, and of the emotions in her lyrics.)

At one point, I had to admit to myself that I might never be able to finish Data Sketches if I kept on having writer's block. So I started thinking through other projects I was already working on, that could be a compelling last project for Data Sketches. And I realized that the final project I was working on at ITP was as good of a candidate as any.

For the last few years I've been wanting to bring my data visualizations into the physical world. I've been obsessed with the idea of expressing data-driven stories as installations that people can actually walk through, to explore and touch and feel. Once I realized that the biggest reason why I couldn't design any projects in the physical world was because I work in the 2D, and have no idea how to think in the third dimension. As soon as I realized that, I created Legends with Three.js and WebGL. That project really helped me learn how to think about space, lighting, and camera position.

Bug being able to code in 3D isn't enough to create a physical installation, I also needed to learn hardware. In the fall of 2019, I did an Artist-in-Residency at New York University's Interactive Telecommunications Program (ITP), a master's program that is at the intersection of art and technology. I went to ITP with a big hope: to create a physical data installation in collaboration with talented people.

I audited Introduction to Fabrication where I learned about power tools and machines and played with the laser cutter, and Introduction to Physical Computation where I learned about circuits and sensors and lights and Arduinos. And through those classes (and the office hours I was asked to hold as a resident) I was able to meet my two project partners: Christina Dacanay with her beautiful illustrations and bold uses of color, and Tina Rungsawang with her architectural background and keen awareness of space. I knew they would be perfect complements to my data and code background.

When I asked them if they would do a final project with me, they immediately agreed (and I was so happy). We decided to aim for an installation around women in computing to be exhibit at ITP's Winter Show.

week 1 | data

From the very beginning, we agreed that we should center the project around women in computing. It's a topic that I've been very interested in since I did Legends with women Nobel Laureates, and Tina and Christina were in complete agreement that it'd be a very fitting pursuit at ITP (which combines code and computation with art).

I started my data gathering the same way as Legends, with a Wikipedia article. I was able to find a page titled Women in Computing, and from there I discovered the Wikipedia category Women computer scientists. In this first attempt, I wrote a script that requested the page, used JSDOM to turn the HTML text into JSON, and iterating through that for the women's names (I've done this many times for previous Data Sketches projects also, whenever I had to scrape a webpage). I then used wikijs to get more detailed information from each woman's Wikipedia pages, including number of backlinks, references, and links in the page, as well as their basic information and summary.

Once I had the initial dataset of 416 women (it was really cool to see so many women computer scientists listed), I wanted to see if there was something interesting for us to center the installation around. But I had to first filter out any women whose pages were stubs and didn't have basic information. For the 247 women who remained, I took a look at their distribution of birth years, and number of backlinks and references. Nothing was out of the ordinary, so I decided to concentrate on the women with more than 100 backlinks that may have more fleshed out Wikipedia pages (I thought backlinks might be a good proxy for completeness).

With those 29 women (which included Tamara Munzner!), I took a look at the intersection of their backlinks, links in the article, nationalities, alma maters, common words in their summary...the only things that stood out were that majority of their nationalities, when listed, were American (which only showed that this list is unfortunately skewed towards America). I was hoping that clusters would appear, like if some were known for their HCI work, or for their backend optimizations work. But those patterns didn't emerge, most likely because of the simplicity of the data I decided to pull (just their summaries instead of their whole articles) and because of the simplicity of my own analysis (I was merely counting intersection of keywords in summaries).

Instead of trying to devise more sophisticated analysis (we were quite limited on time), I decided to take a step back and try my hand at another dataset. I had initially avoided scraping the Women in Computing article since the women's names were scattered throughout the paragraphs instead of in a list, and it was much less straightforward to scrape. But once I realized that the data was much richer there - it succintly described each woman's accomplishments and even gave an approximate timeframe for most women's contributions - I knew it'd be worth the trouble.

For this second attempt, I wrote a script to:

  1. Use wikijs to access all the links in the page, and filter down to only those that are links to other Wikipedia pages
  2. Use Google's Knowledge Graph Search to determine whether the Wikipedia page is for a person or thing, and only keep those that the Knowledge Graph returned "Person"
  3. For each person, go through their Wikipedia article summary and see if it includes "she/her" or "he/him", keep the ones that had more mentions of "she/her"
  4. If there were no mentions of either, mark the person for a manual check
  5. Save name, summary, and certainty of gender into a spreadsheet
Once I had the spreadsheet, I went through each person to double check their gender. For each of the remaining 46 women, I wrote a one sentence summary and found an approximate year for their contributions by reading through the Women in Computing page and cross-referencing their own Wikipedia pages. I also in the process highlighted approximately 30 women that I found particularly inspiring and brilliant.

Because of the space constraint for the Winter Show, we could only design an installation with a maximum of 20 women. For the final pass, Christina, Tina, and I all sat down and narrowed the list down to the 16 women that inspired all three of us.

week 2 | sketch

This is probably the most different and most interesting sketch section from all the other Data Sketches projects I've ever done. It should more accurately be called "prototype".

For our very first brainstorm, we asked ourselves: what would we want to make if we had no restrictions on time, space, and money? We went through many different ideas, but at the core, we agreed that we wanted to highlight the invisibility of these women, and raise awareness for their incredible contributions in computing. We wanted to create an experience that would have people leave feeling inspired, stimulated, and wanting to go through it again. We also wanted to create something where people's interactions with the piece can alter it, and they could maybe even leave something of themselves behind.

The last point of participants being able to affect the artwork was the most important to me, as I've always loved work like Domestic Data Streamers' Golden Age Art Installation where people are invited to add to the artwork and change how it looks at the end. I love that it turns people into participants, that it gives them a sense of ownership in the artwork.

For the second brainstorm, we brought in installations we pinned for inspiration, and a lot of the aesthetic we seemed to converge on were installations with many small objects hung and suspended in mid-air. We especially liked Judith G Levy's Memory Cloud, where nostalgic photographs are placed within small souvenir viewers. People are invited to walk through dozens of these viewers, and peek in to see each 35mm slide.

We loved the idea of people interacting with each object, and decided to map each object to a woman and put their information inside. We wanted people to take their time reading about each woman, and I really liked that looking into the viewer felt like looking into a secret world that we previously had no idea about.

From there, the rest of the data mappings came quickly: the women would be organized in a field by the year of accomplishment, from earliest to most recent. And their height from the ceiling would be by the number of their backlinks. So the more backlinks and more "renowned" they were, the higher up they were because they were more "out of reach", and the less backlinks the lower down they were to be more easily accessible. This was especially important to us, to promote those women that were less well known. We wanted the interaction to be that a person would pull the object down from above their heads and read inside the object. We thought that interaction could also be how we keep track of which women were interacted with, how we give user feedback, and how we potentially affect the end artwork.

To have people be able to leave something behind, we decided to have a "summary board" placed at the end of the installation with each woman's name on it. We wanted to prompt people to place a star sticker (hehe) by a name that they had never heard of before, but were inspired by after the installation. (We initially wanted our experience to have a clear entry and exit, with the summary board placed by the exit.)

Because this installation was also the final project for the Physical Computation class I was auditing, there were regular check-ins on the idea and scheduled "playtests". In our first idea check-in, our teacher gave us the feedback that it'd be really nice that walking up to and pulling down that first object would also trigger other objects to drop down. That feedback inspired us to cluster the women into groups, so that when a person walks up to the first woman, the others in the group would drop down and create a path to guide them through the installation.

Once we had the initial idea fleshed out, Christina went about sourcing materials, and Tina and I went about trying to figure out how to detect a person interacting with our object. We brainstormed several ideas, including a pulley system that would complete the circuit when pulled down on (thus sending a signal), a motor at the top that would unwind the cord when pulled on, a capacitive touch sensor on the object, an infrared motion sensor placed somewhere near the object...all the options felt overly engineered, overly finicky, or overly sensitive.

When we went to our Fabrication teacher's office hours, he suggested we go simpler. He gave us a vibration sensor (the circuit completes when we shake the sensor), and a tilt sensor (the circuit completes when it's tilted up) to experiment with. As you might be able to tell from the gif, the vibration sensor took quite a bit of force to trigger, which meant that if we put it in our object the person would have to really shake it (instead of gently picking it up) for us to know that they've interacted with it. On the other hand, though the tilt sensor wasn't perfect (there didn't seem to be an exact angle that it started sending "on" signals, and even then those signals were quite noisy) it was a much better option.

(We also considered an accelerometer, which would be more sophisticated than a tilt sensor and be able to tell us the angle of the tilt, but we ultimately decided against it because it was ~4x more expensive than the tilt sensor.)

Through this process, we also realized how finnicky a motor would be to deal with and fragile it would be especially with the thousands of people coming through the Winter Show and pulling on the cords (with many kids!). Even though we really liked the idea - motion always really captures attention - we decided to nix the idea. Instead, we decided to put lights in the object since changes in light are still quite noticeable and even more importantly, much easier to control.

We immediately liked the metaphor the lights gave us: we decided to have the hanging objects all start out dim - a metaphor that these women are largely unknown - and as people went through the installation and read about each woman, their light would grow slight bit brighter. We hoped that by the end of the show, all of their lights would be brightly lit.

For the lights, we went with Adafruit's brand of individually controllable RGB color lights called the Neopixel. We originally went with this because we thought we might try encoding data into the color of the lights as well but decided against it because it was quite hard to distinguish most colors from each other unless they were very different (like between, red, blue, green). We decided to continue using it even though we weren't going to have different colors, because they gave us better control over brightness than normal LEDs.

Unfortunately, when we first ordered the Neopixels, we ordered 100 individual ones, not realizing we'd have to solder each of them to their own boards so that we could wire them up to the arduino to communicate with. I learned a very valuable lesson that first time I soldered one of these Neopixels, that they have four pins (one for plus, one for minus/ground, one for data in to receive commands from the Arduino, and another for data out to send the commands to the next pixel if we ever decided to chain them) very very close together and they CANNOT touch. And if they touched, they would short-circuit. (Trying to hold down the wires so they wouldn't move as I very carefully tried to solder the pins to a cut-up board.)

I did four more of this. Unfortunately, two of them shortcircuited and we decided it really wasn't going to scale up very well. On the bright side, I eventually learned how to use a multimeter to test for shorts, instead of trying it out on my (and Tina's) Arduino boards and shorting them...

Fortunately, because there are very many talented and experienced people always on the ITP floor, they were around to advise us on using a Neopixel strip instead, since it has a much nicer and larger area we could solder to. Unfortunately, those larger areas were still quite close to each other and if we weren't being careful, ran the risk of the solders running into each other. Amidst those struggles, I was introduced to soldering paste, which makes it so that the solder only sticks to the conductive part and doesn't stick to anything else. That was such a time saver. And after soldering the wires to the strip, I learned to use a hot glue gun to make sure that the wires wouldn't fall off easily.

(These sorts of lessons are precisely what I went to ITP for πŸ‘) At the same time we were figuring out the electrical components for each object, Christina was sourcing and experimenting with materials. She quickly landed on using fillable Christmas ornaments, and we went about acquiring one from a nearby Michael's to prototype with and ordering three dozen from Amazon at the same time. Christina then went about sanding the nobs off of each ornament, and carefully drilling holes into the center for wires to run out of. To hide the tilt sensor and Neopixel within the orb, we first tried to spray paint the interior of the orbs. We had bought reflective spray paint that was supposed to give a mirror-like finish, but it didn't turn out well. That was the night we learned reflective spray paint only works well on glass and not plastic πŸ˜… We also took the opportunity to test out two ideas we had: whether we wanted people to peek in through a small hole to read the information, or whether we should leave the orb as is and have a larger area to read the information from. It was really great to have the spray painted prototypes, because right away we could tell that the peephole would be way too small and too much of a struggle for people to read large amounts of information through. (The two options we considered for how the orb should look like)

We also realized that we should place the information placard at a certain angle so that people would 1. know that there was a placard inside they needed to read, and 2. be incentivized to pick it up in the correct direction to trigger the tilt sensor. (Prototype of the second option with no peephole, and a large transparent area to read the information through)

It was really fun to prototype the materials, and the whole process was so different from working digitally (where gathering materials is not a thing). Overall it took about two weeks for us to settle on all the materials, and we had to be really strategic about buying just enough material to test with without being wasteful if we decided against the idea, and ordering the materials online early enough that it'd get to us by the time we needed to start production. I also learned a lot about prototyping with cheap, readily available materials (like masking tape, cardboard, and tin foil) to mimic the effect we were going for just to be able to quickly try out an idea.

Since the reflective spray paint didn't work out, Christina suggested cellophane (Christina was an absolute pro when it came to materials). This not only hid the tilt sensor and neopixel, it also had a beautiful effect when lit up in the dark. (A quick prototype with cellophane and the information printed on vellum, a translucent paper to help mimic frosted acrylic and diffuse the light, that can also be printed on.)

week 3 & 4 | code

Once we had all the physical components for the orb determined, I went about with the coding. To control the tilt sensor and Neopixel we settled on, we used the Arduino. Arduino is a brand of microcontrollers (I think of microcontrollers as small, simple computers that we can use to control other devices), and the way we use it is by wiring the pins on the Arduino (circled in orange in the photo) to our lights, sensors, motors, etc. Each of those pins have a corresponding number it is identified by, and so in our code, we can tell Arduino that a certain pin is being used as input to read from (usually a sensor) and another is being used as output to write to and control.

(You never thought you were gonna get a run-down of Arduino from this project, did you? Well we aim to go above and beyond πŸ˜†)

Because this was my first full project with Arduino, I started my code super simple. I used a button as a stand-in for the tilt sensor, and an LED in place of the Neopixel. I used an array to keep track of which pins the buttons were wired to, and another array for which pins the LEDs were wired to. I also had an array for keeping track of which buttons were pressed, and another array for each LED's brightness. Every time the button was pressed it was the equivalent of a "tilt" and I'd increase the brightness of the LED and save it to the brightness array. I wanted to use the button and LED (which I was already familiar with how to code) first so that I could get the functionality working first, before having to worry about incorporating the tilt sensor and Neopixel - both things I haven't worked with before. (Four Neopixel strips and one orb with Neopixel and tilt sensor all frighteningly wired up for my code testing purposes.)

Once I had the basic functionality, I went about replacing the LED and push button with the Neopixel and tilt sensor in my code. The Neopixel actually comes with its own library of functions, and because it assumes that I'm programming a strip of many Neopixels, most documentation examples involve instantiating the Neopixel class with an array of pin numbers. I actually had to do some gymnastics and guesswork to setup each orb's Neopixel individually. (Next time, I might consider using something else, but for this project it was already too late to look for other options.)

Reading the tilt sensor was a little more straightforward, though I did have to adapt an Adafruit example to implement debounce on the reading. That was probably the most interesting lesson I learned about working in hardware, that nothing in the physical world is clear cut like it is in computers, and all sensor readings are noisy.

With the Neopixel and tilt sensor code incorporated, the rest of the code from my first iteration worked perfectly. (I got this working around 1AM after a few late nights and I was so deliriously happy.)

With one orb working, I had to make sure it could work scaled up and with groups. We had decided to group our list of 16 women into four categories - Mathematicians, Computer Scientists, Creatives, and Others (executives, inventors, etc.) - and if any one orb was lifted up, the rest of the women in the same group would also light up in a delayed stagger. (Working out the logic for when an orb is picked up and all the other orbs in the same group should light up in a staggered pattern, and the opposite happening when the orb is put down.)

To implement the delayed stagger, I needed to incorporate time into my code. The only way to do this in Arduino is with mills(), which returns the number of milliseconds since the Arduino began running. I refactored my code to store group information: whether the whole group should be lit up or not, which orb in the group triggered that change, and at what millisecond that change was triggered (which I called startTime). I could then use that information to loop through each orb in the group, and depending on which orb triggered the change (activeOrb), calculate each orb's startTime (when its own brightness would change) based on the distance of that orb from the activeOrb. I would then set the orb's brightness if the time was past that orb's startTime. (The Neopixels lighting up when the tilt sensor in my hand is lifted, and dimming down when the tilt sensor is down.)

I also ended up coding in a fade animation by storing a start and end brightness for each orb, which made the brightness change much gentler, and added in code for a pulsating introductory orb to catch attention and explain the installation.

Coding in Arduino was an interesting challenge in trying to make sure I was being memory efficient, especially when I had to add in groups and remember milliseconds (which were stored as unsigned longs). It definitely made my code more convoluted, but I had a really fun time trying to figure out the challenge.

week 5 & 6 | production

This was the point at which I had finished my fellowship and had to go home to SF (and then to a conference in Tokyo), so I only was able to help Tina and Christina finish prototyping the orbs and wasn't able to help them through the production process. I highly recommend checking both of their writeups (Christina's and Tina's coming soon) for more details on this section.

One of the first things that Tina did to kickstart the production phase is to 3D model an enclosure to hold both the tilt sensor (which is very sensitive to the angle of its placement) and Neopixel in place. We also received a brilliant suggestion from our teachers to use a phone jack to connect our orbs to our wires, so that we could make the orbs easily detachable. That way, just in case an orb malfunctioned, we could easily separate it and debug it away from the rest of the installation. Tina also incorporated the phone jack into her enclosure design, and together with Christina, 3D printed 17 of these enclosures and spent roughly 30 hours soldering them all. (3D model of the enclosure.) (Tilt sensor and Neopixel soldered to the phone jack within Tina's 3D printed enclosure.) (The orb wired up and working as a stand-alone piece!)

They also designed the information within each orb, and laser etched and cut it into acrylic. And finally, they translated the data I gave them into positions and heights in the space we were given for the Winter Show. They then used planks of wood to hang the orbs from the ceiling. (Install timelapse)

It was absolutely incredible to see this whole project come together. It received great feedback at the Winter Show, where over two hundred people experienced it. My favorite stories are of women, young and old, reading through each orb and looking inspired and emotional, placing a gem in the groups they most identified with. (A board where we asked people what groups they identified as - definitely not the most optimal Venn diagram, but probably much easier to understand for non-dataviz people.)

I'm so grateful to Tina and Christina for being the best teammates to pull off this ambitious project (in 6 weeks!), and to all the teachers and residents and students who graciously offered their time and wisdom. I don't think we could have pulled this project off anywhere else.

And I'm so happy and grateful that this is the project that I get to end a whole decade with, to end Data Sketches with. I think it perfectly encapsulates my personal journey - of going from someone that was purely technical, to learning more about design, to finding and chasing and executing on my dreams of doing a full data installation. I'm excited that this ushers me into the next phase of my career, where I want to do more art, and bring more data visualizations out into the physical world for more people to get immersed in and enjoy.

Thank you for three years, Data Sketches πŸ’–

Shirley