Motivation and Goals

Intralogistics is characterised by complex material flows and a high degree of variety. Processes such as commissioning and error handling are currently heavily dependent on the use of qualified specialist personnel – an expense that is rising further due to a shortage of specialists and long travel distances. The research project Semantic Commissioning of Mobile Robots therefore pursued the vision of intralogistics in which robots can understand their environment in context and interpret logistical situations independently. As shown in Figure 1, a mobile robot explores its environment using SLAM technology, whereby a 2D laser scanner captures geometric structures and an RGB camera provides image data for semantic analysis. The aim was to use foundation models to generate a semantic understanding of the environment that would enable semi-autonomous commissioning.

Approach

The SeI_MoR pipeline was developed for implementation, which merges geometric and visual sensor data and processes it into a semantic map. The image data is segmented using the Segment Anything Model and classified by a multimodal large language model (LLM). Not only object classes are determined, but also their mobility (static, semi-static or dynamic). Combining this semantic information with the geometric SLAM map enables the creation of a context-aware environment map. In addition, a human-in-the-loop mechanism ensures continuous quality improvement of the classifications.

Results

In extensive tests at the Mobile Robotics Lab, the developed pipeline was able to demonstrate its performance under realistic conditions. In a complex test scenario, a classification accuracy of 86% for object dynamics and 66% for object classes was achieved. Particularly noteworthy is the ability of open vocabulary classification, which makes it possible to reliably recognise even previously unknown objects. Feedback from industry confirms the high practical relevance: semantic maps can significantly accelerate commissioning processes, improve navigation and greatly facilitate the integration of mobile robots in dynamic environments.