Leveraging AI for Automated Complex Freight Handling

Welcome, and thanks for joining us today. I'm Shirley Goh, and this is my colleague, Lauren Miller. Today, we're talking about leveraging AI to improve the complexity of freight in automated unloading. And as we go through this webinar, if you have questions, please feel free to enter them in the Q and A function. So a little bit about us first. I'm Shirley Goh. I work in product marketing at Boston Dynamics with a focus on warehouse automation. Previously, I also worked at MIT and Boston College in their research spaces. And prior to that, I was a staff writer and editor at The Boston Globe. Lauren, can you tell us more about what you do and your background? Sure. So my name is Lauren Miller. I lead the autonomous capabilities team at Boston Dynamics. I've been in the robotics field for a little bit over fifteen years now. I started in academia. I did my PhD at Northwestern postdoc at UC Berkeley. I worked at a Bay Area startup doing last mile logistics for a number of years, spent some time working on autonomous mining and construction vehicles, and then have been at Boston Dynamics for the past six years. My team at Boston Dynamics works on perception, machine learning, and task planning, decision making, many of the things that make the stretch hardware actually do useful things in the warehouse. So before we dive into AI in complex freight, I wanted to take a step back and talk about the advances that have been made in robotics in the past few years, and what's changed. Why was it so difficult before to automate inbound processes? What wasn't working with more traditional automation? Yeah. So, you know, in a sense, something like unloading a truck seems pretty trivial, like an easy problem. Right? You open it up, there's a bunch of boxes, Take them out and put it on a conveyor. Easy. Right? But trucks and the inside of a truck is actually a pretty complex, diverse environment. When we open the door, we have no idea what we're gonna find inside. Sometimes cases are stacked near neatly. Sometimes it's, you know, chaotic avalanched mess. There's a lot of variation. There's different types of trucks and trailers. The walls are not actually always, flat. The floor is not flat. There might be things hanging off the ceiling. The cases can be damaged. They can be ripped. We can have spills. There's a lot of real variability in the environment, which is challenging for traditional automation. So something that, you know, we've been able to do with Stretch because we are designed for this application. We're a mobile base, so we can autonomously move throughout the trailer. We can move in and out. We have a long reach arm, so we can reach cases up in the highest part of the trailer and be much more adaptive to the environment. And we carry our perception system with us. We have sensors integrated into our platform. We don't rely on a lot of fixed infrastructure. So, you know, we can we can adapt on the fly to changes in the environment and to different type of cases in the trailers. So big step up from, you know, traditional bolted down fixed automation you might see in a factory or in a manufacturing setting where you can kind of preprogram the same task and just do it over and over again. We're much much more adaptive on stretch. Okay. And I also wanted to clarify. I feel like sometimes there's this misconception that, you know, our robot stretch is just an industrial arm affixed to an AMR, but it's so much more than that. Right? Like you said, there's the perception system. There's the manipulation. It's really one streamlined robot. Right? Yeah. So so we've designed the entire platform from the ground up, the base as well as the arm, the perception system. So it all does work together, and that lets us, you know, do things like coordinate our reach and our picks with base motion so we're able to move more quickly and respond to to different environments a lot more effectively. And like you said, we have our perception system on board, so Stretch is constantly observing the environment around it and can adapt to changes. And, you know, we also do have it's not just the onboard system. We have off board systems as well. We have dashboards that are communicating metrics to us, to customers, and we're also integrating with, other parts of the warehouse. So Stretch will command the conveyor that it's placing to it. We can change the position of the conveyor so that it's, you know, following Stretch as we're unloading or, you know, removes and retracts autonomously. So it's it's much more than just, you know, arm bolted onto a base. So let's start with our current state. Stretch has been deployed for a few years now. It was a first mover in the automated unloading space, and others have followed. Where are we in terms of its capabilities and what it's doing right now? Yeah. So Stretch is out there doing real work right now at many different customer sites. We're deployed broadly across North America, Canada, and Europe. And we're moving, you know, around five hundred thousand boxes a week. And when Stretch is moving, the rate and reliability is quite good. So some of the things that our customers appreciate is, you know, we're able to work full shift. Our battery life is is more than your average shift in a warehouse. When stretch is out there, it is delightfully boring a lot of the times, so we love to see that. Yeah. Our customers have mentioned that sometimes stretch actually works so well that downstream, they can't keep up with it. Maybe, for example, they have people palletizing manually. Can you tell us more about this complete inbound solution we're offering? Yeah. So we're excited to offer a multi robot solution, where Stretch will place cases on a conveyor. They will travel down the conveyor, and then the solution will sort by SKU and build, you know, one or two pallets at the end. So we are excited to, be able to offer customers complete solution and eliminate the the bottlenecks that we we see with some of either either manual palletizing or, other automated solutions that maybe don't integrate quite as well with ours. We are always trying to keep the downstream automation in mind. So while Stretch is unloading a truck, you can configure how we place cases for either manual or downstream sorters. So you might need to place, you know, label up for certain downstream automation. So we do a lot on the Stretch side to make sure that we're integrating with, downstream automation or manual palletizers, whatever their system requirements are. And then the other piece is we will, Stretch is designed to be a multi application robot in the warehouse doing many different tasks related to case moving. So end of line palletizing is something that we are also targeting as as an application for Stretch. So that means we can offer customers a full inbound solution, including trailer unloading and downstream automation end of line palletizing. So in the few years that automated unloaders have been on the market, where are they in terms of capability? What is it that they do well, and where is it that they run into challenges at this point? Yeah. So so I mentioned that when stretch is when stretch is cranking, it's it's the we're moving really quickly, and we're reliably able to unload trailers for customers. We do really well for many of our customers who are in the apparel or footwear space. So our sweet spot is cases up to fifty pounds. So we we do handle heavy cases and, you know, between six inches and thirty six inches. So we we have a we have a we have a solid range where we're performing quite well. We're putting a lot of development right now into expanding the complexity of freight that we can handle both within those ranges and expanding a little bit beyond. So, I mean, we can handle small cases and large cases right now, but we might do so at a reduced rate. So we wanna make sure that we're always pushing, getting our performance and our our pick rates progressively higher. Some other aspects of complexity that we're working on is the variability. So how many different cases you might find when you open a door and and you start working on a trailer. And stretch can handle high variability. You just might see lower pick rates and potentially more human intervention. So it's something that we're we're working on actively. Maybe you can describe kind of the spectrum of complexity that you find in a trailer. For example, what does it look like from one extreme to the other? Where you see really simple freight, maybe not so many SKUs and everything is orderly, to more types of SKUs and things are a jumbled mess. Yeah. So there's there's a couple different axes of complexity. One is variability. So how many different SKU types you might see in a trailer. Simplest being there's a single a single SKU in one trailer. So it's all the same box end to end. Somewhere in the middle is every each facade is a different SKU or every two facades or three facades, you have a transition. And then all the way on the difficult end of the spectrum is, know, every single item is different from every other item. The other aspect is order or disorder in the stacking pattern. So and they they tend to be related, but you might have very neatly stacked facades where everything you know, imagine a brick wall or something where people where where everything is stacked very neatly. Or, you know, you open the door and it's just a completely disordered pile of boxes that you have to deal with. So those are kind of the two axes of complexity that we think about. And are there cases where boxes are really wedged in, where maybe a container comes in overseas and they're packed really tight? Or maybe the boxes are stacked at, like, different depths? Does the robot struggle with those? So that's a pretty subtle and and interesting point is you might you might have cases that are stacked really neatly, but that might still be difficult for an automated solution. And so being packed really tightly and being wedged in is a perfect example of it looks really structured, but you can't, you know, simply pull a box out. You might have to shift it to the left or or do something, a little less straightforward to get cases out. The other consideration is when we start to have variations in-depth. So we tend to unload trucks from front to back, so you would pick the entire front facade and then the the facade behind it. But if you start to have cases where the depths of the cases are such that you have the the facades are interlocked interlocked with each other, you can't follow that type of picking pattern. And so you might have to pick boxes that are actually behind the cases you're currently picking in order to pick them. So the pick order gets a lot less straightforward when you have certain stacking patterns. So even with these challenges when when things are wedged or with the stacking, I feel like for people, this kind of, Jenga situation is not that big a hurdle, and it's a matter of you pick this up and you put it down. Why is that so hard for robots to do? Yeah. Jenga is a great analogy. You know, like, my my four year old can do it. Why is this so hard? Right? Part of it is that people, have much richer sensory information than any robot or automated system does. So when you're playing Jenga and you're prodding that middle piece to see if it'll move, you're you're looking at what's happening, you're using feedback from your finger, you're using the positioning of your arm, and you're using years of experience built up playing Jenga or doing other physical things in the in the environment. Even the robots with the most complex perception systems still have less rich information than that. So those those actions, like nudging and responding really quickly to subtle subtle cues in the environment, are are hard for automated systems. Another challenge, particularly in the the cases we were talking about with either wedged cases or when those facades are interlocked with each other is that the the only information you have is the front face of of the cases. Right? So you can't necessarily just go off of the the visual information you have. You actually have to start interacting with the boxes to figure out that you might have to, you know, pick pick one box before you can remove another one or nudge nudge something over to the side, or you might not be able to lift straight out. You have to lift you know, push to the left and then lift out or something like that. These are all these are all things we do really intuitively without thinking that hard about it. But getting an automated system to to do those sort of adaptive human like motions is is a challenge. Alright. So how do engineers actually go about expanding the complexity of freight that robots can handle? So a lot of it is the unglamorous part of robotics that's, you know, observe, test, iterate. We have fleets running at customer sites, and we're constantly, watching and paying attention to where the robots slow down, where they hesitate, where they ask for help. We build our own evaluation infrastructure at home, so we rely on simulation. We rely on setting up physical tests that mirror some of the situations we see on customer sites and, you know, just just working on them until they just iterating on them until they work. Our customer partnerships are a really important part of this, so there is no way that we can ever recreate the variety and level of complexity that the robot actually encounters on customer sites. So we do a lot at home to try and mirror those real scenarios, but we also rely heavily on our customer partnerships. So we will do beta tests of software, where we go, and we, will use newer versions of the software to see if we've solved specific customer problems. We also rely on customer feedback. So there's infinite axes where we could improve our performance on Stretch. We've talked about a few of them, but there are others, and we rely on our customers to, you know, help guide the most impactful ways that we can improve the software. I've heard that in some cases, customers send us their freight to help train the robot in it. And also that we sometimes send our teams to train the robot on their sites as well. Because as much as we try to replicate the freight and the conditions, the trailers, the conveyors, we can only do it to a certain degree. Yep. Can you tell us more about that? Yeah. So, we have our own test facility that we rely on heavily for testing and development, but it will never be the scale and the variety and the complexity that we see on customer sites. So we work very closely with our our customers and our partners. They will send us freight. So we've got a couple of trailers or smaller box sets from our customers that help us focus on real problems that matter. And then we will also do targeted software testing. So we will send our team to customer sites where we might test newer versions of the software for a week and do rapid iteration cycles with our customers to make sure that we're focusing on the right problems, and nothing is better than field testing to validate your software. We also don't try to solve everything at once. We will we might focus on better handling of small cases in one release. We might focus on heavier cases in another, and, we might try to get our rates up on the interlocked type or wedged cases in a different situation. So the other thing, that particularly in the types of scenarios we're talking about where you have wedged cases or, these jangle like situations where you need to nudge or or move certain boxes to get to other ones is the newer AI driven methods, foundation models, VLMs, VLAs, vision language models, or vision language action models. The difficult cases for complex freight are perfect applications for these these newer models. So advances in the field are huge for for Stretch's ability to expand the types of trailers that we can handle. So let's talk about AI. This is the topic that everyone wants to discuss right now. Where does that fit into Stretch? Yeah. So there's there's a lot of enthusiasm right now over AI or learning based methods in manipulation, and we are also really excited about them on Stretch, particularly like we've been talking about for applications like handling more complex freight. Very excited. Huge amount of potential, but there is still a gap between where we are right now with many of these methods and the rate or reliability that we need for Stretch. And that gap is closing quickly. At the same time, Stretch is at customer sites right now. And so when we're thinking about rate and reliability, we're trying to shave milliseconds off of our pick rates and get, you know, intervention rates down. So we are hyper focused on performance and reliability for our customers. Our approach on Stretch and the way that we get these learned systems out in the world doing real work at the rates and reliability that we need to support our customers is we get the robot out doing the task, even if it's a simpler version. So on Truck Unload, like we've been saying, we we tend to do really well for cases that are within our wheelhouse and that are maybe stacked more neatly and aren't wedged in boxes. So we get the system out there using a maybe a mix of more traditional engineering methods. We are collecting data. We are learning about what matters to our customers. We understand the task really well. That gives us the data that we need to train these learning based methods, and we understand the problems that we need to solve. So on Stretch, we will you know, we'll always be focused on rate and reliability above all else, and the learning based methods will let us expand to some of those harder cases. So tell us more about that. How are you balancing the adoption of riskier technologies and supporting general customer needs? So we're in a really unique spot, on stretch being a warehouse robotics platform, but also being part of the Boston Dynamics team. So we, in house, have the Atlas project that is cutting edge of these learning based methods, generalized mobile manipulation platform. And so they are, you know, pushing the frontier of general purpose human level manipulation mobility moving super fast, and we're learning from that work constantly. So what Atlas is doing is, you know, directly affecting what we start to do on Stretch in some of these more challenging environments where you need that human like behavior of nudging cases or or playing Jenga in the environment. So we're in kind of a unique space that we are not a company that is solely focused on warehouse robotics that has a single platform, but we get to be in house with projects like Atlas or Spot that are ahead of anyone else in the field in terms of generalized AI and human level manipulation and capabilities. Atlas also will learn a lot from Stretch. We've been operating in the warehouse for years now. Atlas is also looking at automation tasks in the warehouse. So they'll be able to learn from data that we collect and tasks that we're learning both for other tasks in the warehouse, and then potentially that translates to different activities Atlas does that are, you know, basically material handling. So at Boston Dynamics and on stretch in particular, we are still pragmatists. So we do use a lot of good old fashioned engineering where it does enable us to perform predictably and reliably. So in many cases, our happy path for Truck and Load when boxes are stacked perfectly neatly, the task is, you know, pick place, pick place. And that is something you can you can hand engineer and it works really quickly and really reliably. The skill right now with where we are with learning is knowing when to utilize more of these traditional engineering methods and learning based method learning based methods. And Boston Dynamics has a long history in robotics, and that gives us the judgment and the skills to know when we should be using newer learning based methods and when we should be using traditional based methods and how to transition between the two. And that'll change over time, and we will adapt. So you've mentioned the customer piece and the fleet data piece. How big a deal is that, especially compared to, you know, other automated unloaders on the market? Yeah. That's huge. And there's a couple different ways that both we're using fleet data and that we enable our customers to use fleet data. So right now, our Orbit platform gives customers a way to understand what their automated fleet is doing. So we capture images as Stretch is unloading at fixed frequency. So customers can go in and see what's going on, why did the performance why was there maybe a performance issue or just understand more about their freight than they did before. And then at the same time, we're able to use that data internally to continuously improve what we're doing on stretch. So, you know, particularly with some of these learning based methods, getting the right models and the right algorithm is part of it. But the important part for scale and reliability is getting the the scale of data you need. So we have that pipeline on stretch already built. And then the other piece, you know, is is really understanding what problems actually matter, which we get from just being out there and doing the task and getting that fleet data insight. So can you give us an example of how traditional and more model learned systems would complement each other? Yeah. We're using machine learning for perception. We have been for a long time. That is state of the art. Learned based systems, you know, work better than hand coded systems for doing things like detecting boxes. So that's been in our our stack for a long time. What we're investing in now is using learning based methods deeper in the software stack. So getting more into decision making about which box to pick next or the manipulation itself, what your extraction strategies should look like when you're picking a box out of a facade that's very tightly very tightly stacked or wedged. And, you know, we don't we don't necessarily need AI to tell us how to pick cases on the happy path where you have perfectly neatly stacked boxes, then, you know, the hand the hand engineered pick top down, left to right would work perfectly well. But as we extend to these scenarios where you have complete disorder or these wedged interlock cases, that's where where we're leaning into learning based methods. So we've been talking about truck unloading for most of this conversation, but robots that are doing automated unloading are starting to do more than that and move beyond unloading. Can you tell us more about where that's going? Yeah. So Stretch is best known as a trailer unloading platform. That's our flagship application. We've been out there doing it for years. But we are a mobile manipulation platform designed for general case moving tasks in the warehouse. So we are looking at case picking as our next major application. In that scenario, Stretch is gonna get to come out of the trailer, and we'll be driving around large warehouses, picking from under racking, and building up pallets that will then be part of an outbound solution for customers. This is a, you know, more more complex application in a lot of ways, large largely that we have to build up that navigation, functionality. But a lot of the work there actually does translate from trailer unload. So many of our learnings on how we pick and place cases and the manipulation part of it, we can directly apply to the case picking application. What parts of truck unload do you think will translate to these new applications? So specific examples might be, you know, how we perceive the particular cases or items that we're manipulating, whether they're in a truck or in pallet bays under racking for customers, would directly translate. And the more that we are leveraging these learning based methods that allow us to do more generalized pick and place tasks, the more every new application becomes a lower lift because we're building off of data and experience. And, you know, at the end of the day, we're we're picking cases out of stacks of boxes, whether it's truck unload or picking from a pallet and case pick or doing depalletizing at the end of the line. And then we're placing them into a different configuration, whether it's onto a conveyor or end of line palletizing or in case pick. So those basic pick and place tasks will will generalize to other applications. So we expect that every every new application we do, we'll be able to do a little bit more quickly. You know, beyond just stretch and warehouse automation, there's a broader version of this at Boston Dynamics. We have three platforms, stretch, Spot, and Atlas. And we're increasingly thinking about the AI underneath them as a shared backbone. So data from Stretch unloading trailers can inform Atlas doing different tasks in a warehouse or in even different environments. And similarly, data collection that we're doing for Atlas or for spot manipulation might improve our capabilities on Stretch. So, you know, we're really excited about the promise of more general purpose manipulation capabilities on all of our platforms. Okay. So before we go to questions, I wanna come back to where we started. There's so much excitement right now in the industry about AI and humanoids and what will be possible. For someone who's running a warehouse today, what do you think their takeaway should be? So, you know, I think the the value, the value of stretch is tied to how much of a customer's freight that that we can handle. And we're doing a lot of work for our customers today. We talked a little bit about where stretch is is performing really well so far, but there's a lot of opportunity in freight complexity that will unlock even better ROI for our customers. So, you know, interlocked loads, higher SKU variability, disordered trailers. Right? Closing the gap is is major focus for development right now on stretch and and the biggest lever we can pull for for customer ROI right now. The proximity that we have on Stretch to the rest of Boston Dynamics is a huge benefit for our customers here. We can leverage, you know, cutting edge techniques to push the boundaries of what an automated solution like Stretch can do. At the same time, we are hyper focused on rate and reliability for our customers. So Stretch will continue being reliable and focused on performance and doing the tasks that our customers need us to do. That's a really good place to land. Don't go away. We're taking a break for just one minute. And when we come back, we're going to answer some of your questions. Here at Arvato, we work closely with our customers to fulfill and solve a lot of the challenges that they have with their logistics and supply chain operations. We had a vision for completely automating a carton from loose loaded in a container all the way to the outbound process. We already had the scanning and systematic pallet building and sortation. Stretch was kind of that last puzzle piece that we were missing to have a true automated loose load unload solution. Stretch can unload everything that we bring in in a loose loaded form. It allows us to improve the back end, the sortation, with just a little bit of our logic change, and we're able to really achieve higher throughputs than we ever imagined. Welcome back. So we've been going through some of the questions that have been submitted. And Lauren, if it's okay, I'll just run through some of them with you. Sure. So, one that has been asked a few times, can Stretch handle polybags? Not yet. So Stretch has a, planar vacuum gripper. We can actuate individually actuate the cups, so we we we are pretty adaptive in what we can handle. Not polybags yet. We know this is important for our customers, but right now, we're optimized for, cardboard cases primarily. So no no timeline promises, but we are we are thinking about it. So we're getting a lot of questions about the ROI of the robot. That's a really common question, and I can take this one. With optimal use, at least two shifts a day, the typical ROI for stretch is two to three years. And, obviously, with advancing the complex freight part of the piece, we'll see even greater ROI. Here's another question. Does Stretch need a manifest? Does it need to know what's in the truck before it starts? No. So so pretty much you open the door, drive Stretch in, and press start unload. Stretch has a perception system on board, so we are constantly, taking images of the environment and adjusting on the fly to the SKUs that we encounter. So no no manifest needed upfront. We're also getting asked about whether stretch can handle mixed loads, more than one SKU. How many SKUs? Yeah. So so we do handle mixed SKUs and variability quite well. So, we can handle cases of of different sizes, you know, up to fifty pounds. Six to thirty six inches is roughly the dimensionality we can handle. We handle brown cardboard cases, but we can also handle graphical images on boxes, you know, no problem. Stretch will adapt its planning on the fly if it encounters a box it hasn't seen before. So we handle all of that, as is. You might see some of what we've been talking about here is at the really high end of complexity. You may see reduced reduced rates or, higher interventions, and that's a lot of what we're working on improving with, the complex freight related work. But we still do handle many different SKUs in the trailer today. So one person is asking about whether Stretch collects data while it's unloading and whether customers have access to that. So through Orbit performance dashboards, I believe they have that. What kind of metrics are customers seeing, and how are those beneficial to them? Yeah. So there there's a number of metrics that that we share with our customers, which is useful in understanding the performance of stretch and then might also be interesting in just them understanding, what they're getting in their trailers. So, you know, we we share CPH, our our pick rates, intervention rates. We can also provide ins insights into things like dropped boxes, how often we're able to multi pick. So that's one stretch is picking, more than one box at a time, and that that really helps our our performance. The other thing that customers can see is, via Orbit, our fleet management dashboard, we will show images that that the robot is actually taking over the course of unloading a truck. So that provides additional insights for customers both on some context for how the robot is performing and then maybe just general insights for what their trucks look like that they might not have had before. And the robot's capturing those images pretty frequently. Right? So they're able to kind of pinpoint what exactly those trailer conditions are when they see something of interest throughout one workflow. Is that right? Yeah. So so providing those images helps our customers understand how specific localized scenarios in the trailers, so how boxes are stacked, anything else going on in the specific region of the trailer impact Stretch's performance. Alright. So we have time for one more. There's some concern among people that AI is overhyped, or about the humanoid AI bubble. Should customers be concerned if our warehouse robots are adopting some of these unproven methods? So no. I guess on the one hand, you know, I don't think that these methods are overhyped. It maybe is a question of when we get to the type of rate and reliability that you need for the type of tasks that Stretch is doing. Stretch is more of a specialist than a generalist. But I I expect that, the AI methods that we're leveraging will pan out. It's just a question of of when. And and when I say when, that's that's talking about, like, true generalizable AI. Right? For what we're talking about on Stretch, we're still trying to solve fairly specific problems, like extracting wedge boxes or responding on the fly to a dropped case or or some other scenario that we don't have to hand code in the same way. So we are solving still much more targeted problems. And I mentioned before, you know, we are we are hyper focused on the rate and reliability that our customer needs. So where we add AI will be in expanding into, capabilities that we don't currently have, and that will not mean that we regress on our existing levels of rate and reliability. We will we will only see more capabilities and more adaptability. Yeah. Lauren, I've got three more fun questions for you. What's your favorite Boston Dynamics legacy robot? Oh. I have to say handle. It's part of the stretch lineage, and it is just such a fun robot, and it is so dynamic. And that's that's really boss dynamics and just seeing that big battery swing around. Okay. Beautiful. Now, this is a hotly contested topic on the stretch team. If stretch were to wear a hat, does it go on the mast or the gripper or the base? Team mast, for sure. And why is that? It's it's the head. Right? It's where, like, the perception is. It's where the lights are. It's on the top. Yeah. Yeah. Last question. Favorite robot movie. This could be any robot from, like, Johnny five all the way to Wild Robot. So I I like horror movies. So probably ex Machina or two thousand one Space Odyssey, which, you know, may maybe that's why I like developing robots to avoid avoid that future. But I think those are fun. Okay. Well, that's all we have time for today. Thanks so much for joining us. Thank you, Lauren. For more information about Stretch, please visit our website.