Google Innovation Grant: The Mind Palace

Quartz was awarded a Google Innovation Grant to create an interactive VR or AR game. We specifically wanted to make an interactive experience that enhanced a story, so Michael Tabb and I worked in tandem to create a game.

The process of creating this on a 4 week timeline while working on other projects was intense, but I really enjoyed it. It required a level of elastic thinking and problem solving that you only really encounter with new technology.

There was A LOT to figure out.

We had to figure out: would we build our own app, or could we try to fit it into the existing Quartz app? What program would we use to build an app? Which one of our long form journalism projects would be appropriate for this platform? What great AR experiences have been created in journalism? Who is our intended audience? How can we make sure that users can enjoy the project via Android, iPhone, and desktop? What do we want this to look like? How can we make sure that the project is small enough to be opened on a phone using data?

The “Memory Palace” was popularized by the show Sherlock as the way that he recalls information. This is a real technique that plays on human psychology that allows people to remember long strings of numbers. It converts arbitrary lists of digits into stories and scenes, which are easier to remember because they're visual and emotional.

AR technology is not standardized across platforms, and doesn’t work in certain browsers, so we ended up creating two separate, but related experiences: one for iPhones and one for Android/desktop users. Above is the Android/browser experience. If you can’t see it, it’s probably blocked by your pop blockers and/or privacy browser extensions, so please switch those off momentarily while you scroll, rotate and click through the project; it’s made to be interactive.

I used the grant funding to take a crash course in Cinema 4D, and applied that training to creating a stylized 3D experience. I art directed, modeled and textured the objects to fit within the Exceptional Humans (paywalled video series) framework.

The concept was to recreate the “Memory Palace” popularized by Sherlock. You take a string of numbers and assign names to them, and create a story. In the Android version, I put the string of numbers on one side, and once you rotate the model, you see the story played out in 3D. Nonsense stories are encouraged, it tends to make them more memorable.

This particular story goes as follows: a BUN jumps over a STICK, knocking another BUN into a GATE. Overhead, a HERO jumps over TWO TREES and lands on a VINE.

The iPhone version (only playable within Safari) is playable here.

Ultimately, I think we created an excellent prototype that would be a great memory improvement tool if it could move through multiple randomized strings of numbers, rather than one example. If you’d like to see the full project, it’s here (paywalled).

Opioids Documentary

In fall 2019, Quartz released a Gracie Award-winning opioids documentary: a sprawling, international look at how Purdue pharmaceuticals have marketed opioids in the U.S., and how they’re currently using the same tactics to market them abroad under Mundipharma. My team traveled to France, the Netherlands, and Pakistan to report this story, talking to black market pharmaceutical dealers, doctors, and drug reps.

I was in charge of post production while juggling art direction of two other video series. In this documentary, I was responsible for art direction, animation and finishing edits.

---

Organization and Tone

The goal of this investigation was to create something that was about opioids but…funny. We didn’t want to make another crushingly depressing documentary and were going for tongue and cheek. It was a difficult balance that we achieved with high contrast footage that emphasized shadows, contrasted with bright, almost cutesy graphics. We added a voice actor that served as the main character of the documentary and had him narrate as if he was an opioid marketer. We told him to come in with a “used car salesman” suit, and we shot stills of him in the studio. From there, I chose selects and then used Photoshop to turn his brown suit lavender, putting him on a bright blue background to give a late-night-infomercial feel.

After Charles’ styling was decided, I moved onto organization. I created a 1000+ cell spreadsheet that included thumbnails of the graphics, checks for completion and fact checking, and notes on changes. It became our post production bible that we used to create daily schedules and estimate final delivery. Because of the check marks and color coding it was immediately apparent what needed to get done at a glance, and keep track of any changes.

---

Animation

My proudest creation from this project is my pill monster. It represents the workings of the pharmaceutical industry and how drug reps, insurance companies, doctors and the black market work in tandem to create this demand for new drugs. My interpretation of the “pharmaceutical machine” was personifying steampunk gears, money, pills and syringes into a scuttling monster that travels across the globe. I wanted it to be a slightly more sinister Howl’s moving castle, but also have an inherent element of absurdity.

---

Documents

Half of this story is told through legal proceedings, so I had to figure out a way to make documents interesting. Most of these documents were not high resolution, so I ended up using a mixture of roughen edges and an alpha texture on top of a masked paper background to bring text up word-by-word and give the animation dynamism.

I always used a crumpled paper texture to give documents a bit more visual interest and did a lot of layering so it wasn’t just the title page of the document, but some of the contents.