Ethan Lauer, a software engineer, answers your questions about robot perception, world modeling, and what spooks our Stretch robot. Read the interview below and watch Ethan discuss how Stretch sees the world.

Working at Boston Dynamics

What is your role at Boston Dynamics?
I’m a perception engineer on the world modeling team for the Stretch robot at Boston Dynamics.

What is world modeling?
Just like how humans use our eyes, ears, and sense of touch to understand what’s around us, the robot needs to do the same thing. Stretch has a bunch of cameras and sensors, and the robot uses that information to process and get an understanding of what’s around it. So we can then manipulate objects like boxes, and move around obstacles as well.

What does one version of a typical day look like for you?
I check in with our customers to see how the robots are running. It’s great to see all of the boxes that our robots are moving for them. It’s very inspiring to see the work that we’re doing here is having a real life impact. It also provides great feedback for how we could improve the robot as well. I usually take that inspiration and start doing some mad scientist brainstorming on a whiteboard with my team, or writing code for a new feature to be able to detect obstacles, or running the robot and seeing if what we did helped fix the problem and help improve the robot.

Getting into robotics

What first sparked your interest in robotics?
I was about 11 years old. I really felt that spark. I was on a behind the scenes tour of one of my favorite movie sets, and we had entered the creature creation lab and the special effects lab. And I was absolutely amazed by how all the creatures and characters that I knew and loved were just created by motors and wires and a couple lines of code. From then, I knew that robotics is where I should be. And around that same time, the Boston Dynamics Big Dog videos were insanely popular on YouTube. I remember seeing that, and thinking maybe one day we could make these robots that not only look cool on the screen, but actually are helping people in real life.

What was your first impression the first time you saw a Boston Dynamics robot?
My first day at Boston Dynamics was the first time I had seen a Spot robot or any Boston Dynamics robot in person. My new boss had opened the double doors into the lab, and I felt like it was Willy Wonka opening the doors to the chocolate room, and I was Charlie Bucket. I was absolutely dumbstruck to see the Spot robots just kind of roaming the hallways. It was a little bit like meeting your childhood hero, to be perfectly honest. And having studied robots in school, I knew how complicated it is to create that kind of a system. I was blown away by the ingenuity and how it’s very casually just here and in the world.

What’s your favorite thing about working at Boston Dynamics?
Boston Dynamics is the most creative, innovative, and supportive company I’ve ever worked in. Everyone here is at the top of their field and at the same time, everyone is learning from everyone as well. Even though we work hard to bring our deliverables in, we are not afraid of our silly side. There are a ton of Spot robots that have dog toys around them or googly eyes on them. Me and my coworkers will laugh when the robots fail catastrophically, and that’s one of the joys of seeing the bloopers that we show on YouTube because it’s all very real, and it’s a very grounded and creative place to work.

What advice do you have for aspiring robotics engineers?
In high school I was very physics and engineering focused. I was going to be a mechanical engineer, and my engineering teacher encouraged me to move into robotics. I ended up going to Worcester Polytechnic Institute for robotics engineering. I would say if you are passionate about robotics and if you really want it, go after it and get it. I did not expect when I was younger that I would work in robotics at all, let alone Boston Dynamics. That was a pipe dream way down the line. And a few years later, here I am, and I’m extremely grateful for that. Also remember to have fun with it because that’s part of robotics. It’s hard work, but it is also incredibly rewarding and fun.

How Stretch visualizes the world

How does Stretch “see” its surroundings?
Stretch sees the world with a variety of sensors. The robot has two cameras on its perception mast, which has standard RGB images as well as depth clouds. There are four sensors on the base of the robot; those are lidars that give you a 360° planar view around the robot so it can detect when there’s boxes and walls around it, and navigate around those obstacles in order to pick boxes. There are also several distance sensors inside the gripper. This is used to refine its sense of where the ceiling is and get a better understanding of where some obstacles are. It uses all these to get an understanding of what’s going on around it, and where it’s picking.

How does Stretch plan a pick?
When Stretch tries to go and pick a box, it will take a picture and decide based on the box’s location where it can pick without interrupting the rest of the boxes. Just like a human would not pick from the middle if they were playing a game of Jenga, Stretch would pick from the top first to make sure it doesn’t cause an avalanche of boxes to fall. Once Stretch decides which box is most optimal to pick, then it will figure out the best way to grab that box. Based on how heavy the box is and what its dimensions are, Stretch will figure out which side to best grasp the box from. Stretch will look at all its surrounding areas to make sure it won’t collide with anything on its way there, and it’ll plan its trajectory for its arm to go pick that box.

What is one low-tech problem that our high-tech robot has encountered?
Even though Stretch is a very, very high-tech robot, it turns out it can be a little afraid of spiders, actually. When I was working on testing the robot one time with several of my coworkers, we kept seeing Stretch run away from what seemed like a ghost, just kept driving away, driving away, driving away. And in troubleshooting it after many long hours, we found there was just a very fine spider web that was woven on top of our base lidar that was causing the robot to think that there was an obstacle in front of it when it really wasn’t. And it was pretty incredible that this high-tech robot could be spooked by just a little bit of a spider web.

Is Stretch pre-programmed, or working in real time?
Stretch is making all of the decisions on how to move the boxes from the trailer onto the conveyor in real time. There are several parameters that we use at sites so it can understand where the dock is, very general ones, but everything else is pretty much plug and play, and our perception system takes care of the rest. Stretch uses all the information it sees of where the conveyor is, where the walls are, and is able to model the world around it. Same thing goes with the boxes — at first encounter Stretch doesn’t know what kind or size or how heavy the box is. Stretch is able to automatically pick those boxes and learn from them.