I joined the Google Research team Cerebra in 2016, when machine learning was rapidly advancing into new areas of utility in our lives. The research team was tasked to figure out how to take the latest technological innovations and make them useful and accessible, even invisible. I worked on using a variety of biometric inputs to improve security of Android OS phone authentication, as well as enhancing the overall use of the Pixel phone camera (e.g. speed, automation, editing, and curation).
In the summer of 2016, our team (Seattle’s machine intelligence and research team at Google) was developing an alternative phone OS, rich in machine intelligence capabilities. The effort was eventually folded into the Pixel 2 and 3 OS, and a number of features made it into development (Top Shot and Google Lens was at one point demoed on the Pixel landing page). I was a supporting designer that built out flows and prototypes for the camera, especially related to improving photography and video capture, as well as on-camera object/product recognition.
We ran through multiple sprints, brainstorms, and double pyramid vetting sessions (where engineering and design come together to find the most compelling features based on user desirability and engineering feasibility), to drill down to a few different features to explore deeply. On-camera detection of objects survived the culling.
Recognizing items on screen presented a few challenges. How do you provide an on-screen affordance that did not block the view of their viewport? We designed a way for the camera to signal to the user that it recognized something with a row of moving dots at the bottom of the screen that corresponded to the x-values of various key points of the identified item. These dots would hint to what exactly on the screen was being identified, without requiring the user to turn on the feature first. This would make the feature more discoverable and useful when users discover spontaneously different instances in which the phone could identify items for them.
Once the user tapped the row of dots, the dots would then rise up to frame the identified object, and tapping on the the object’s framing dots (or the area it circumscribes) would reveal more information about the object, such as a related website (e.g. for a storefront), trivia (e.g. for a statue or plant), links to shop (e.g. for a book or product). Additional use cases could be digitizing or translating text. See below for mid-stage explorations of the feature.
Amid the hubbub of developing a new mobile OS, we also explored how machine intelligence could change the way we captured and reviewed photography and video. Our team was already working on the Google Clips camera, which could film for hours at a time, identify the most interesting clips, and present them back to you in an easy-to-digest format. How could we translate that technology into a mobile phone? Do users need to tap the shutter to start taking photos or videos anymore? From usability tests of Google Clips, many felt that they wanted a button for better control, to make sure that certain moments were captured. As a result, we considered supplementing user-controlled capture with automatic capture, and the OS would present alternative captures that may have been better than what the user captured, whether due to human error or unpredictability of events (like a child smiling).
I engineered this prototype to feel what it would be like to tap anywhere on the screen to capture an image.
We also came up with UX flows for when the camera was able to identify someone in the shot, and suggest sharing it with a related third party (like a child’s parent). We also considered how a user might open the camera app, whether that could be inferred through certain movements of the accelerometer, or by a gentle but unequivocal squeeze of the phone’s hardware. We also explored what it would be like for a camera’s shutter to be activated by tap on any part of the screen (instead of a single red button or portion of the screen). I engineered a simple prototype (see right for the video of the working prototype).
During my stint at Google's research and machine intelligence unit (led by Blaise Argüera y Arcas), we worked on a number of rapidly evolving projects. One of them was a new phone concept, which ended up folding into the Google Pixel. During its conception, the majority of the team understandably (though predictably) rallied around the theme and technology of machine intelligence.
Though I was a junior designer, I was increasingly concerned that the mainstream applications of machine learning technology in and of itself (as impressive as it can often be) ultimately doesn't solve many of our most urgent human problems. It can make everyday interactions more efficient, but the efficiency gains at the time didn't seem all that impressive, especially when Apple's iPhone was already loaded with machine intelligence.
What if, instead, we delivered a product unlike any already on the market? Inspired by Tristan Harris's original internal Google deck on digital distraction, his new foundation, and The Miracle of Mindfulness by Thich Nhat Hanh, I developed the concept of the Mindfulness Phone.
Fortunately, Blaise was excited to embrace contrarian and alternative ideas, and had already set up a monthly meeting called the Brain Trust, where anyone in the organization could present any idea or problem to the engineering leadership for review. I presented first to my design team and afterward to the Brain Trust.
I presented it to the Brain Trust, and received rave reviews from the design team. A number of ideas throughout the deck (such as the "intentional search bar") influenced the final designs for the machine intelligence OS for the next Google Pixel.
In addition to this initiative, I traveled to Berlin with a team of Google researchers to explore how user privacy could be another strategic differentiator and value proposition for the next Google phone. (Though Google leadership was reluctant to adopt privacy for their devices, Apple ended up making it an increasingly important pillar of their brand strategy over the last decade.)