Robot-based bin-picking is considered to be the supreme discipline of robotics and has long been a focus of pioneering work at Fraunhofer IPA. Thanks to machine learning, the outstanding capabilities of this technology have now been extended and innovative features have been added.
A robot technology at Fraunhofer IPA with a history – that is bin picking, for example. After all, the separation of bulk goods is a task which arises in nearly all industrial productions and is predestined to be carried out by robots. The task is monotonous, physically demanding and cost-intensive. Although automated solutions for separating goods exist, such as vibratory bowl feeders, they are usually not flexible enough. Despite this, the use of robots for bin-picking tasks is anything but widespread – quite the opposite! This is precisely why experts have been carrying out research in this field for many years, optimizing the technology and, in particular, solving user-specific problems.
Back in 2007, Fraunhofer IPA filed its first patent application for the object pose estimation methods it had developed. Now, even in 2023, some challenges still have not been solved. Because although the task may sound simple to outsiders, it is extremely difficult for a robot. The high economic potential of the application – a ROI within two years in three-shift operation is quite common – is one thing, but the requirements placed on robot-based bin-picking by series production is quite another. Consequently, of the more than 200,000 new robots sold worldwide each year, only a few thousand are used for bin-picking. The majority of robots, on the other hand, grip blindly or use 2D machine vision at best for semi-chaotic delivery tasks, such as depalletizing.
Artificial intelligence (AI) and, to be more precise, AI Picking, have now given the application a considerable technological boost, thus enabling known problems to be overcome. Bin-picking cells are the first link in a connected production or assembly line. The timing of such a connected line is dependent on each station delivering a guaranteed output.
“Typical” bin-picking involves two uncertainties. Firstly, it cannot be guaranteed that a robot is able to remove all the parts in the bin. If necessary, the last remaining parts must be separated by hand. Secondly, the cycle time increases significantly the emptier the bin becomes. This is because the object recognition system may have problems reliably detecting parts lying on the bottom of the bin. The resulting fluctuations in the cycle time can be compensated for in two ways: either by assuming a worst-case scenario or by using buffers. The entire line must therefore adapt to a potentially high cycle time, or the bin-picking process has to start earlier and develop a lead in order to avoid downtimes. These uncertainties currently prevent the widespread use of bin-picking in practice.
More autonomy, better cycle times, greater robustness: These are the added values that the use of machine learning (ML), a subfield of AI, offers for bin-picking. The current AI Picking application has gone through several preliminary development stages. Fraunhofer IPA has been integrating ML into existing technologies since 2017 with its first project “Deep Grasping”. The aim of this project was to improve the self-configuring capabilities of the robot system, i.e. to increase user-friendliness, while at the same time making it less prone to error. With this in mind, a virtual simulation environment was developed where artificial neural networks could be trained to recognize objects. These networks were then subsequently transferred to the real robot.
The project results enabled further milestones to be successfully achieved. For example, Fraunhofer IPA hosted the “Object Pose Estimation Challenge for Bin-Picking” at the IROS 2019 conference. The aim of the challenge was to estimate the position and orientation of chaotically-positioned objects using different training data and methods and to compare them. IPA also provided a set of training data. In the next research project, “Deep Picking”, a completely new disentangling technology was developed which allows the robot system to detect entangled parts and to plan a way to separate them and to pick them up individually.
Last but not least, in the “Sim4Dexterity” research project, a simulation environment is being created in which robots can learn handling and assembly manipulation skills with the help of AI in a time- and cost-saving manner. In the course of their research work, the IPA experts found a way to use the developed technologies for more than just bin-picking. By further developing them, they are also capable of detecting product packages, which is important when it comes to palletizing and depalletizing processes in logistics.
AI Picking has already been successfully evaluated in a wide variety of use cases on the IPA laboratory demonstrator and with initial pilot customers. The demonstrator merges the aforementioned preliminary work, thus taking bin picking to a new level. Last year, the technology was presented at two trade fairs for the first time; several industrial implementations are currently being planned. We would welcome further partners interested in implementing this technology.
As far as bin-picking is concerned, a lot has happened at Fraunhofer IPA over the last 15 years or so. The current situation is a decisive step towards “automation of automation” – which the institute is strategically striving to achieve – in other words, towards self-adapting and self-configuring automation systems.
With AI Picking, the institute offers a self-configuring simulation solution, for example for estimating the position of parts or automatically defining gripping poses and selecting grippers on the basis of the CAD model. This significantly reduces the amount of expert knowledge required to set up the application or teach in a new object. Setup times are cut by up to 84 percent. The data processing time for object localization, gripper selection and gripper pose determination is extremely fast at 20 milliseconds. Mixed bins can also be emptied automatically, and special functions such as segmenting packaging material or disentangling parts are also available.
The current focus of research work is on virtual feasibility studies. These can be carried out purely by simulation, thus giving users information about the feasibility of automating a bin-picking solution very early on in the planning stage. The team headed by group manager Richard Bormann would like to offer such cost-effective feasibility studies for a wide range of bin-picking scenarios and, in the future, also include automatic optimization and automatic gripper design in its portfolio. An objective industry benchmark for bin-picking systems is also planned.
Join us - Challenge at ICRA 2023!
Fraunhofer IPA is organizing a challenge at the prestigious ICRA 2023 conference, as it did at IROS back in 2019. This “Virtual Manipulation Challenge” aims to reduce the gap (also called Sim2Real gap) between the simulated and the real robot application. Participants can first train robot behavior in the simulation. The effectiveness of the developed methods is then also evaluated in a simulation with numerous standard and exceptional scenarios. The Challenge includes three application scenarios: (1) removing single parts from chaotically filled boxes, (2) screwing in and inserting parts in assembly, and (3) stacking boxes and packing 3D free-form objects. Thanks to well-known industry partners providing the use cases, participants have the opportunity to implement and test their own ideas using real-world and complex scenarios.
Fraunhofer Institute for Manufacturing Engineering and Automation IPA