Fraunhofer Institute for Manufacturing Engineering and Automation IPA
Fraunhofer Institute for Manufacturing Engineering and Automation IPA

“Sort with me!” – An interactive, self-learning handling robot

© Fraunhofer IPA
The hands-on and interactive exhibit uses a simple sorting game to demonstrate that tasks which are easy for humans to perform are in fact extremely difficult for robots.

In short

The handling robot developed at Fraunhofer IPA presents new and sometimes sophisticated AI technologies for handling tasks by means of a simple interactive game for young and older players about sorting objects. The demonstrator is controlled via speech recognition and is mobile, which makes it highly versatile.

In detail

“Sort with me!” is an interactive sorting game with a self-learning and interactive handling robot. Via its sensors, the robot recognizes objects placed in front of it and sorts them according to different criteria unknown to the human player. In the next step, the human player has to guess what these criteria are and use them to sort other objects correctly. The winner is the first player to identify the secret sorting rule and communicate it to the robot. The aim of this game is to demonstrate various everyday challenges that intelligent robots must be able to overcome both today and in the future - including visual object recognition, handling objects of all kinds, and voice interaction with users.

When it comes to visual object recognition, robots must be capable of “learning” new objects in real time and recognizing certain object properties such as color, surface texture, or shape. The demonstrator evaluates these properties and uses them to determine sorting rules in the game. To achieve this, the Fraunhofer IPA researchers apply machine-learning methods such as Siamese networks and integrate different sensors.

Unlike humans, robots find it very difficult to grasp unknown objects. Therefore, the experts have developed a model-free gripping technology for the handling robot, i.e. one that functions without the need for a data basis. Instead, a Convolutional Neural Network (CNN) is used to evaluate the gripping quality of various access points. As a result, the demonstrator is able to pick up known as well as unknown objects placed at random in front of it by the user without any problems.

Such a game becomes particularly entertaining when you can interact directly with your opponent. Thanks to neural networks and appropriate speech output, the handling robot automatically recognizes the language spoken, enabling it to ask questions, give instructions, and reply freely to questions posed by the user.

Insights into the project

© Fraunhofer IPA
The user can interact with the robot with natural language.

This might also interest you