This article was written by Krista Shapton and Andre Brueckner. Krista is the UI/UX designer for Stretch and previously worked at Kuka, Rethink Robotics, and Vecna. Andre is a product manager for Stretch, and previously worked at Rethink Robotics and Universal Robots.

Our robots are complex feats of engineering, performing backflips and self-righting after a fall. But learning to operate Stretch, our mobile case handling robot, isn’t complex at all. With high turnover rates for personnel, the long term success of Stretch with our customers depends on their ability to train new staff quickly. Alleviating this recurring challenge allows customers to maximize the use of Stretch across all shifts in their operations. Creating an easy user experience for operators was so important that our code name for revamping the interface was The Simple UI. This post dives into what the new user interface looks like, how we got there, and what’s next.

The Simple UI

A demo of the redesigned trailer unload UI.

Starting an unload with Stretch is easy. The operator is guided through a series of steps on the console, including a trailer preparation checklist, pairing the robot to the workcell, and tapping a play button to begin unloading. The UI leverages clear calls to action and images, and breaks the process up into simple steps to take guesswork out of the equation. Users don’t need to enter any information about the types of boxes or trailers since Stretch handles this autonomously once the robot begins unloading. New operators learn how to run Stretch in about one hour and become fully certified in two days.

A troubleshooting screen shows boxes backed up on a conveyor belt with instructions on how to proceed.
A troubleshooting screen that walks the operator through fixing the issue.

Once Stretch begins working, the UI displays key performance metrics on screen: number of cases unloaded, cases per hour, and total runtime. If the robot stops for any reason, operators are shown exactly what caused the stop and how to resolve the issue. The console allows the user to view the progress and speed of other Stretch robots on site as well.

A demo of the end of trailer feedback animation.

Usability and simplicity were table stakes for Stretch’s interface redesign, but truly compelling products don’t just work well, they are also a pleasure to use. As we hone Stretch’s capabilities and ease of use, we also build in moments of delight. When Stretch finishes a trailer, a Pac-Man-inspired animation scrolls across the console, celebrating the unload. More delightful moments are in the pipeline as Stretch carries on Boston Dynamics’ hallmark of merging playfulness with world-class performance.

The Design Process

Our goal when remaking the interface was for Stretch to be as easy to use as a home appliance like a dishwasher. (And unlike other industrial robots, Stretch’s ability to work autonomously with minimal input made this possible.) We went back to the drawing board, starting from a core list of design principles: simplicity, accessibility, reduced cognitive load, system status transparency, recognition rather than recall, and guided error recovery. Everything from the color palette to individual buttons were specified in a new design system, which was built as a reusable component library. In parallel, the workflow was overhauled, using an iterative design approach that incorporated feedback from customers, product management, and engineering.

Once we had a first draft of the new UI, we needed to test it. Would users balk at a longer workflow with simpler steps? Would they accept checking boxes instead of just reading instructions in a list? We hit the road to answer these (and more) questions at customer sites. To make the test as realistic as possible, we created a touch prototype that looked and behaved like the real truck unload workflow, embedded in a portable operator console. We asked operators to perform tasks like starting the robot and turning on the multipick feature, and A/B tested different interaction flows to uncover their preferences, learning curve, and overall impression of the experience.

A hand is poised to tap a "continue" button on a console screen, which is propped up on a table with a backpack behind the console for stabilizing.
Our test setup at a customer site, showing the clickthrough prototype on a works-like version of the console (sans robot).

Bringing warehouse users into the design process not only answered our workflow questions, it gave us a host of new ideas to explore. Each conversation revealed unique insights: from excitement, to frustrations, to learning how our UI may be interpreted by users with dyslexia or color blindness. In the end, we were able to pre-validate the UI redesign, and make a number of improvements before its release to customers a few months later.

What’s Next

Here at Boston Dynamics, we continuously build, test, learn, and improve. And on our Human Robot Interaction/Industrial Design team in particular, our mantra is making complex robots simple to understand and intuitive to use. With Stretch, we were fortunate to start with an amazing robot that takes care of most of the hard work itself, only requiring a few inputs from the user to do its job. With the trailer unloading UI caught up to the robot’s high standards, we turned our sights to improving the troubleshooting workflow. A new suite of tools provides operators and service engineers more transparency into troubleshooting events on the robot so they can be triaged and resolved quickly. We also brought support ticket tracking and data monitoring widgets into the user interface. These tools help customers team up with support technicians when help is needed.

A rendering of a pendant that resembles an Xbox controller.
A rendering of the new pendant that will be available with the next software release.

What’s next for Stretch’s user experience? We’re in the final stages of releasing a completely redesigned pendant, which is used to drive Stretch into place before a workflow is started or to its next container. It comes with an ergonomic overhaul that allows hands of all sizes to comfortably and intuitively drive Stretch. The main changes include a larger safety switch, optimized button layout, and weight balance.  It will also feature an updated UI with an improved driving experience and extra tools for support technicians.

Lastly, with Stretch’s next software release, the UI will be translated into six new languages (German, Spanish, Korean, Dutch, Polish, and Ukrainian). Those additional languages will widen access to Stretch, just in time for our upcoming launch in Europe. At Boston Dynamics we’re changing your idea of what robots can do – in lowering barriers like language, designing a simple interface, and sprinkling in a bit of joy, we want to maximize the number of workers who can benefit from Stretch and discover how easy and fun the robot is to use. Stay tuned for more delightful improvements as we continue to push the boundaries of intuitive use for industrial robots.