Hamburg Port Authority (HPA) not only keeps Germany’s largest seaport and Europe’s largest rail port running, but also looks after real estate and infrastructure in the Port of Hamburg with its approximately 1,800 employees. This includes the many bridges in the port of Hamburg, including the striking Köhlbrand Bridge, which is about 3.8 km long. The second longest road bridge in Germany, it has been the most important east-west link in the Port of Hamburg for over 50 years. Around 38,000 vehicles cross the bridge every day. With such heavy use and increasing fatigue damage, it is clear that condition-based maintenance must be carried out with the utmost care. Since 2022, the agile mobile robot Spot from Boston Dynamics has been helping out.

As Head of Research and Development at the HPA, Uli Baldauf is always on the lookout for suitable new technologies for the condition-based and preventive maintenance of the Port of Hamburg’s facilities – drones, AI applications, and innovative inspection robots. This is how he became aware of Boston Dynamics’ agile mobile robot Spot and recognized the technology’s potential for operations at the Port of Hamburg.

We are now at a point where we can have Spot working autonomously in the bridge for several days and nights. Everything is running smoothly: inspections of the defined points of interest and the timing of the mission.

Uli Baldauf, Head of Research and Development

Robot Demo Day at the Port of Hamburg

“At first, I wasn’t quite sure what Spot could really do for us.” To find out, Uli Baldauf organized a demo day with the quadruped at the Port of Hamburg with Roboverse Reply, a company specializing in the integration of robotics solutions. “We were accompanied by our colleagues and potential users of Spot. And then we went into harsh environments: parking lots, green meadows, train tracks, and inside a noisy, dusty bridge, where the cavities included steps and stairs to negotiate.”

During these inspections, the team explored the surveying capabilities of Spot: whether it could carry a lidar payload, for example, which can be used to visualize 3D objects. Roboverse Reply used a variety of payloads, including a high-resolution Leica RTC360 3D laser scanner. It also used the Spot EAP package, which enhances Spot’s autonomous navigation, computing power, and communications.

Kevin Hawryluk, Roboverse Reply Manager for Robotics and AI Topics, who has been with the project from day one, adds, “HPA really wanted to discover the robot’s limitations and gave us several environments to work in. Apart from one surface for large cargo ships by the quay walls, which was too muddy, Spot mastered all the different environments.”

The robot’s capabilities already convinced Baldauf and his team on the demo day. ” Spot was able to walk autonomously and overcome obstacles effortlessly—so we got the internal go-ahead for a longer research project.”

Safety Applications Inside the Bridge

The team agreed that Spot was particularly suited to performing structural inspections in the cavities of Köhlbrand Bridge. Köhlbrand Bridge is already a “smart bridge” with more than 500 sensors permanently measuring values in its interior, reporting on movements and corrosion. But the HPA team also regularly inspects the bridge because the permanently installed sensor technology is not enough.

A major inspection takes place every six years, secondary inspections every three years, and additional walk-throughs annually. Inspection requires up to five people and the work takes months. It involves working in the bridge cavities in difficult conditions. The environment is dusty, dark, and noisy, hot in summer and cold in winter. Strong echoes make direct communication difficult if people are more than 10 meters apart.

The interior of Köhlbrand Bridge is dusty, dark, and noisy, and has to be inspected regularly. (image source: HPA / Florian Steffens)

Welcome Support for the HPA Team

Spot proved to be the right robot for the job. The robot is able to move independently inside the bridge, in complete darkness, and navigate through the narrow passages between the bridge segments. Work that was previously done with a flashlight, pen and paper can be performed by Spot, relieving the inspection team of the unpleasant part of their job.

But when trying out a new technology, it is also always important to bring along the company’s own employees. Uli Baldauf explains: “In the past, when our R&D team approached our colleagues with innovative technologies, they sometimes reacted cautiously because they were either concerned the technology would not really help, or that their jobs could be replaced by machines. That has now changed: everyone was happy that they didn’t need to work in that environment.”

How Autonomously Can Spot Work Inside the Bridge?

“There were two main project phases: Phase One was to answer the question of whether such autonomous inspection is possible with Spot. The answer was a resounding ‘yes’,” explains Uli Baldauf.

“The second phase focused on how to optimize Spot’s autonomous operations in the harsh environment, and whether it was robust enough. What do we need to do, what conditions should we create to leave him there in the bridge for several days allowing him to reel off his inspection program and recharge autonomously as needed?”

Fine-Tuning in the Training Phase

HPA’s use case required custom programming, iterations, and AI training. The Roboverse Reply team came to Hamburg regularly and worked with Spot inside the bridge for three days, eight hours a day. Then it was back to developing and testing new tasks in a replica of the environment for a few weeks at a time (depending on the desired level of detail), before Spot was used again for three days inside the bridge.

New functions and new ideas were tested again and again, to ensure Spot would be successful in real conditions. For example, to get from one bridge segment to the next, Spot passes through a narrow passage with steps on either side. Initially, Spot tried to turn around in the middle of the passage to descend, which was not possible given the narrow aisle. Together with Boston Dynamics’ product development team, Roboverse Reply modified the necessary routines to ensure Spot could handle such challenges.

The agile mobile robot Spot from Boston Dynamics, inside Köhlbrand Bridge. (image source: HPA / Florian Steffens)

Another challenge was that the inspection was carried out in complete darkness, in a monotonous environment that offers very few clues to determine its position, even when illuminated. However, thanks to the sensor technology of the EAP package, Spot can navigate even in these conditions.

Certain POIs are defined in the digital twin of Köhlbrand Bridge. (image source: HPA / Florian Steffens)

Digital Twin of the Bridge

The lidar data in the form of a 3D point cloud and photographs taken by Spot were used by Roboverse Reply to create a digital twin of the bridge. This digital image allows engineers to navigate remotely through an accurate, virtual image of the bridge and zoom in on areas of interest. This can also be done immersively with VR glasses. Critical points or areas, known as points of interest (POIs), can be defined in the digital image. Spot can pay special attention to these POIs during the inspection, for example by taking high-resolution images.

The digital twin of Köhlbrand Bridge was developed by Roboverse Reply. (image source: HPA / Florian Steffens)

Damage Classification

With the digital twin, Roboverse Reply also makes it possible to store information on specific locations in the bridge, which is then displayed to users via an augmented reality solution. This is because every single damaged area and every crack that spot detects is classified according to a system developed by HPA. With each additional inspection of a segment, Spot creates new images of these POIs.

In the digital twin of the bridge, each marked damaged area can be zoomed in and compared with older images. (image source: HPA / Florian Steffens)
In this screenshot from the Hololens application, the HPA inspector has marked two damaged areas in the bridge. (image source: HPA / Florian Steffens)

Reality Check

The aim of this procedure is to obtain reliable answers to important questions: Are there new cracks in the concrete? And are these cracks changing over time? In creating the digital twin, images taken at the same location just six weeks apart were initially compared. The reassuring result: At least over this (relatively short) period of time, no development could be detected.

Thanks to the charging station, Spot can be used autonomously in the bridge for an unlimited time. (image source: HPA / Florian Steffens)

Outlook and Other Use Cases

Uli Baldauf is satisfied with the success of the project: “We are now at a point where we can have Spot working autonomously in the bridge for several days and nights. Everything is running smoothly: inspections of the defined points of interest and the timing of the mission.”

The successful use of Spot in Köhlbrand Bridge raises the question of further possible applications. The port of Hamburg has almost 120 bridges, but other buildings in the port are also suitable for similar inspections. “When we build a new road, for example, the surrounding area is always initially surveyed in detail. Spot could assist the land surveyor’s office,” Uli Baldauf explains. “Actually, the same applies to all surveying work in the port.”

Fire safety tasks are also suitable for automation: The HPA owns large office buildings for its approximately 1,800 employees. Spot could check that all fire extinguishers are in the prescribed locations, fire doors are closed, and lights are switched off. The port also has 300 km of railroad tracks. Spot could be used to check whether the path is clear and that there are no trees or other vegetation in the way.

In addition, paint and insulation samples must be taken at many locations in the port of Hamburg to determine whether they comply with current environmental regulations, for instance whether they contain lead or asbestos. Especially in the case of substances that pose a health risk, this is an ideal task for Spot, protecting employees. It is possible to program an automated process where Spot takes samples with a drill and a suction cup at marked locations.

Spot’s work on Köhlbrand Bridge opens up the perspective of a broader challenge: “Overall, we see great potential in the inspection of bridges with Spot,” Kevin Hawryluk from Roboverse Reply commented. “The Federal Government is responsible for about 40,000 bridges in Germany, and more than 1,600 of them are considered dilapidated. Our project has proven that Spot can inspect them technically. Now in addition to the economic viability, it’s also about the legislative side, permits and certifications.”