Skip to content

Engineering Zoom: Behind the Loupe

by on June 6, 2013

June 06, 2013

In our previous post, A Closer Look at Zoom, we explored the design inspiration behind the loupe tool. But the road from a compelling design to a compelling implementation is often long, windy, and paved with discarded prototypes. While our development team prides itself in not shying away from such engineering adventures, we must always weigh the short- and long-term technical complexities of a design against the value that it brings to the experience. In this post, we’ll share some of the engineering challenges that the loupe presented and how we approached them.

Custom Two-Touch Gesture Recognizers

In early prototypes of the loupe, it became immediately clear that the pinch gesture used to activate the tool would conflict with another key gesture in Paper: Rewind. Rewind is executed by tracing the shape of a circle with two adjoining touches, whereas the pinch gesture that opens the loupe is executed by moving two touches apart from each other. On the surface, these gestures appear trivial to distinguish. However, because the early stages of their execution are often quite similar, we realized that a more sophisticated method of disambiguating them would be necessary.

The problem was that we could not continue building on top of UIPinchGestureRecognizer and UIPanGestureRecognizer as we had been doing. They simply did not provide the fine-grained control we needed in these areas:

  • Activation thresholds: The distance that the touches must move apart or translate before the gesture activates.
  • Mutual exclusivity: Definitively picking the best single gesture to activate given the touch histories.
  • Gesture-specific constraints: Fine-tuning a gesture like Rewind to not activate in ambiguous cases where UIPanGestureRecognizer otherwise would.

To work around this, we first reverse-engineered the built-in UIGestureRecognizers to create custom versions from scratch. We were then able to work off of these to build gesture recognizers for two-touch-pan and pinch that are designed to work in tandem. Not only did this approach solve the problem of disambiguating pinch-to-zoom and Rewind, it also made both gestures more robust in isolation and set us up for a more customizable and nuanced gesture system down the road.

Simultaneous Gesture Recognition

Our key motivation for the loupe-based approach was to give creators the ability to zoom in to add detail and return to the original context quickly. Therefore, we felt it was essential that you be able to work in the loupe while manipulating it. This implied an implementation in which you’d be able to draw or manipulate the loupe with one hand while activating Rewind or the tool tray with the other. In gesture recognition parlance, this means recognizing gestures simultaneously.

Simultaneous gesture recognition poses many challenges, particularly in edge cases. To illustrate, try this: Start a pinch to summon the loupe. Then lift one of the touches (which will leave the pinch gesture active, allowing you to move the loupe). In this scenario, placing a touch inside the loupe should draw, while placing a touch on the loupe’s edge should resize it. A touch outside the loupe should do nothing. If you place two touches at any of these locations and activate the Rewind gesture, none of the previously mentioned actions should take place and Rewind should proceed.

Ultimately, the solution to these many edge cases involved the UIGestureRecognizerDelegate delegating to the various controllers to make localized decisions about which touches the gestures should receive, whether the gestures should activate given the current state, and what to do when the gestures do activate.

Offscreen Gestures

We love taking full advantage of the iPad’s multi-touch system to give a sense of direct manipulation in Paper. Thus, much of the interaction model for the loupe was inspired by treating it like a physical object – one that you could drag and resize intuitively through touch. However, when experimenting with early versions, we quickly learned that ending a manipulation when the touch(es) moved onto the bevel would shatter the perception of physicality.

Much of the interaction model for the loupe was inspired by treating it like a physical object – one that you could drag and resize intuitively through touch.

In the absence of a touch sensor on the bevel, we engineered the loupe to guess at the path of the manipulating touches that go offscreen. This way, if the touches return to the screen you’re able to continue manipulating the loupe instead of, say, accidentally drawing a stroke.

We often find that seemingly small engineering challenges like these pop up whenever we try to push the boundaries of multi-touch. In the case of the loupe, we built it in a way that lets you focus on your work instead of having to focus on not hitting the edge of the screen. If we didn’t tackle these scenarios, the solution would lose its magic.

Page To Circle Clipping

Typical zoom implementations magnify the entire canvas, thereby clipping it to the bounds of the rectangular screen or viewport. Our implementation, on the other hand, clips the magnified contents of the loupe to a circle. Since much of Paper is rendered in OpenGL, it’s only natural for the loupe to also be rendered using OpenGL. Unfortunately, OpenGL has no built-in functionality for rendering anti-aliased circles. Luckily, we were able to solve this problem by constructing circle geometry with a 1-pixel border on the edge that fades the edges and simulates the effect of anti-aliasing.

A larger challenge then presented itself: One of our favorite features of the loupe is that it allows you to work on the edge of the page without drawing at the edge of the screen. (To see what we mean, simply move the center of the loupe past the edge.) Unfortunately, clipping a page with rounded corners to a circle (technically this shape is a superellipse, for the curious) is no easy task.

In the end, we decided to segment the background from the page using a pixel shader. (A nice side effect of this approach is that it gives a crisp boundary to the page no matter what zoom level you’re viewing.) On some generations of iPad, however, the shader proved too costly.

To ensure acceptable performance on all iPads, we first observed that there are three possible states that the loupe can be in: One in which no page edge is visible; one where only a horizontal or vertical page edge is visible; and one where a page corner is visible. We then created separate shaders for each state, and as luck would have it, the more expensive the shader needed to be for each state, the fewer pixels it had to shade: Up to 100% of the loupe has to be rendered when no page edge is visible; up to 50% when only one page edge is shown; and only up to 25% when only a page corner is shown, the most expensive state.

In graphics, you often find that to meet performance goals, you have to defy software development best practices by tossing generality out the door and special-casing key pieces of complex code.

Form and Function

After experiencing the loupe in Paper, we hope you share our delight in its combination of simplicity and utility. While the engineering challenges that zoom presented were considerable, the research and development was well worth it. In future articles, we’ll dive into the custom gesture recognizer work that was introduced in this post. In the meantime, follow the FiftyThree making process and makers at @FiftyThreeTeam. We’re happy to answer any questions regarding what you read here.

We’re always seeking passionate and talented makers who would love taking engineering challenges like the loupe head on. Join us.

Advertisements

From → Excelsior!

Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: