By relying on human perception data, our module can identify interactions with extended reality buttons, grasping interfaces, and in parallel identify improper handling actions or error events during hybrid co-manipulation scenarios. In parallel, digital content augments the physical workstation for intuitive operator support

The Operator Tracking for Co-manipulation (OTC) module serves the purpose of monitoring human activities and preprocessing them for handling inputs. By relying on 3D stereo image data, the OTC module can identify interactions with extended reality buttons, monitor the coordinates of the operator’s palms as active grasping points, identify improper handling actions or error events, and perceive the actual workstation scene for accurate operator support content.

It is evident that within human robot co-manipulation, operator awareness is of utmost importance. The operator needs to be certain about the precise location and method for positioning the fabric, while concurrently receiving feedback concerning the quality of the provided handling inputs.

Under this notion, dedicated human machine interfaces have been designed. These interfaces allow operators to comprehend common goals, describe robot agents’ actions, and easily access safety or performance critical events in an intuitive manner.

Which technologies are utilized and how do the developed modules affect the operator performance?

The Operator Support Module is an application specifically designed to enhance the operator’s experience and performance during fabric manipulation tasks. It involves two applications, the first is hosted on an Android tablet mounted on the robot manipulator, facing the operator. This module provides valuable information and extends the physical workspace by overlaying data and graphics from the digital twin onto the physical workstation. Augmented reality (AR) technology is utilized to create an immersive experience for the operator. By calibrating the relative position of the camera and the tablet within the digital twin, the module can accurately project graphics onto the live image feed from the OTC module’s stereo camera. This allows important cues and instructions to be presented directly within the operator’s workstation. The second application is hosted on an edge computer and uses live image feed from a top-view camera for serving the same functionalities, mostly when the manipulator and the operator are not in proximity.

Extended reality buttons play a significant role in the operator’s interaction with the system. These buttons are displayed as floating circular shapes strategically placed within the workstation. The OTC module monitors the proximity of the operator’s hands to detect interactions with the buttons. In addition to extended reality buttons, the module also offers touch buttons on the tablet’s screen, physical workstation buttons and voice commands for operating the available interaction function blocks.

During the pre-grasping phase, the Operator Support Module instructs the operator by highlighting the intended contour of the fabric on the workbench. This guidance helps the operator understand where to unfold the fabric in terms of position and orientation. Additionally, the module displays the arrangement of grasping points for the upcoming co-manipulation process, clarifying the allocation of human and robot grasping areas. The human grasping points also function as extended reality buttons, enabling a grasping action when the operator’s palms are placed over the designated grasping points for at least one second

During the co-manipulation process, the Operator Support Module provides real-time guidance to the operator regarding the placement location of the manipulated fabric. An augmented floating arrow continuously points to the coordinates where the fabric’s centroid should be placed. If the placement location is within the live image stream field of view, an augmented marker visualizes the exact point. Additionally, a progress bar indicates the quality of the operator’s handling inputs, providing feedback on the effectiveness of the current grasping strategy. This information allows the operator to make real-time adjustments and optimize their approach for improved results.

The Operator Support Module utilizes a score generated by the model-based planner to evaluate the quality of the operator’s handling inputs. This score is represented by a progress bar that changes color from green to red based on the score’s value. A green progress bar indicates high-quality handling inputs and an efficient handling process, while a red progress bar indicates improper handling commands that pose risks. In such cases, the module pauses supportive robot manipulators and notifies the operator to prevent any potential accidents or errors.

By combining the live image feed from the OTC module, AR graphics, and optimization scores, the Operator Support Module facilitates fabric manipulation tasks. It enhances operators’ understanding of task requirements, improves their decision-making process, and ultimately contributes to higher productivity and quality outcomes.

The development of a new, multi-platform, AR human machine interface for fabric co-manipulation tasks can revolutionize the way operators interact with robotic systems. The Operator Support Module, taking advantage of the augmented reality technology , provides operators with valuable information, guidance, and support throughout the fabric manipulation process. By empowering operators to make real-time adjustments and optimize their approach, this interface enhances their performance, accuracy, and overall productivity.

Panagiotis Kotsaris, Research Engineer, LMS, University of Patras

Panagiotis Kotsaris works as a research engineer at the “Robots, Automation and Virtual Reality in Manufacturing” group of Laboratory for Manufacturing Systems and Automation (LMS). He has extensive experience in Extended Reality applications development, web development and multi-modal Human Machine Interfaces.

MERGING’s Response to the Industry’s Need

In the heart of the MERGING project lies our innovative planning framework, designed to address the challenges of collaborative flexible material manipulation. This framework goes beyond traditional methods, offering a comprehensive solution tailored to the complexities of flexible materials. By integrating advanced algorithms and strategies, it redefines the standards of collaboration, and adaptability in the industry.

MERGING introduces a refined planning framework tailored for collaborative flexible material handling.

The MERGING planning framework is designed with a focus on adaptability and realism, aiming to address a range of scenarios from composites to textiles. By abstracting algorithms from specific robot types and geometries, the framework seeks to be applicable across different contexts.

Each module within the framework operates independently, leveraging the modular architecture of ROS. This design choice allows for the dynamic interchange of planning methodologies based on specific scenarios. Integration with our deformable object model is facilitated through ROS interfaces.

During the initialization phase, robots position themselves relative to the fabric. This involves a series of sub-processes tailored to accommodate various robotic configurations, setting the stage for the subsequent collaborative manipulation phase.

The planning algorithm takes into account both the robotic arms and additional planning axes. These axes, essential for reach and positioning, are integrated into the planning process to ensure coordinated movements, especially when dual-arm configurations are involved.

In the co-manipulation phase, the planning algorithms incorporate geometric constraints to facilitate coordinated movements between two manipulators, the rail, and the torso axes. In parallel, the system translates inputs into corresponding robot actions, ensuring that the robots operate in harmony with the human operator’s intentions. Safety precautions are embedded within the framework, and operator-supporting interfaces are integrated to provide a consistent interaction experience. The resulted supportive robot arm trajectories are the result of an optimization process that is characterized by its reconfigurability and computational efficiency.

Movement planning within the framework addresses both translational and rotational movements. A hybrid movement mode combines these two types of movements, offering a more comprehensive approach to fabric handling. The framework also considers additional planning axes, such as linear and rotational, during the planning process.

The final stages of the process, placement, and layup, are crucial. The framework ensures that the fabric is not only manipulated but also accurately placed and laid out on the desired surface or mold. This involves a series of precise movements and adjustments to ensure the material aligns correctly with the target area.

In summary, the MERGING planning framework provides a structured approach to the collaborative manipulation of flexible materials, from initial positioning to final layup.

MERGING’s planning framework offers a solution to the industry’s challenges with flexible materials. Through our innovative approach, we aim to enhance collaborative manipulation, ensuring both precision and adaptability in real-world applications.

Emmanouil Kampourakis, Research Engineer, LMS, University of Patras.

Emmanouil Kampourakis works as a research engineer at the “Robotics” group of Laboratory for Manufacturing Systems and Automation (LMS). Bringing with him a solid foundation in software engineering, simulation models, control algorithms, embedded systems, and electronics, Emmanouil continually seeks to push the boundaries of knowledge in these fields. His research activities include the design of digital twins and simulation environments, cutting-edge algorithms for collaborative robotic cells and a plethora of interfaces utilizing state-of-the-art sensors and actuators.

On top of this, another pressing concern looms over the composites manufacturing industry – the well-being of its workforce. Workers in the composite industry are often confronted with health issues stemming from chemical exposure and physically demanding tasks.

Are safer and more efficient conditions possible in the Composites Manufacturing Industry?

In contrast to common practice of manual labor, MERGING project introduces innovation in the field of robot cognition and dexterity on the manipulation of flexible objects. With this notion, we introduce a novel multifunctional end-effector for the automation of composite layup. Inspired by the industrial needs for efficient and robust procedures, the meticulous design of the end-effector, allows for consistent and collision-free tools utilization, without excess tool changes.

The multi-purpose end-effector not only aims to (semi-)automate composites manufacturing, by addressing a number of operations related to skin and core material handling, but also strives to provide a safe, risk-free working environment for operators.

Introducing a novel multifunctional robot end-effector, to automate the handling of limp materials and manufacturing of composites

This innovative end-effector aims to address these challenges head-on, with keen focus on improving the working environment and elevating performance metrics within the composites layup process. Having one device covering all processes is bringing us one step closer to unlocking the full potential of robotics in composite layup processes.

The device couples all major operations of composites manufacturing and layup, in a single cost-effective device:

1. Composite fabric grippers: It excels at handling of different variants of fabrics, ensuring precise transport and placement. The distance between the needle grippers, or grasping points, can be laterally adjusted with the use of the leadscrew drive mechanism of the parallel gripper. This adjustment of the grasping point distance is valuable for in-hand manipulation or grasping of composites of different widths.
2. Parallel grasping mechanism: It securely grasps core materials, with the use of two holding pads that can mirror like “open” or “close” with the same power input by the intermediate timing-belt powertrain. The pads are designed for achieving friction-based parallel gripping, for core materials, or parallel grasping of tools, such as staple guns, using additional alignment features.
3. Layup rollers: A flat-surface and an edge-surface roller (d.) can independently be deployed for evenly distributing resin during the lamination process. A double-acting pneumatic cylinder is responsible for lowering and retracting each tool. By adjusting the air pressure of actuation air, the stiffness of the tool can be fine-tuned.
4. Dabber tool: A dabber tool is attached to statically flatten layers in challenging mold geometries, like corner edges
5. Grasping of peripheral tools: it can achieve friction-based gripping of peripheral tools like stapling guns or spray gluing guns for performing additional tasks of interest, with the use of the parallel grasping mechanism

The impact of the innovation extends to various industries, including automotive, where it can offer the required dexterity to robot agents, by presenting novel grasping and manipulation skills. Ultimately, this fuels advancements in efficiency, product quality, ergonomics and worker well-being

Papadopoulos Giorgos, Research Engineer, LMS, University of Patras

Giorgos Papadopoulos works as a research engineer at the ‘Robots, Automation and VR in Manufacturing’ group of Laboratory for Manufacturing Systems and Automation (LMS). Holds a Diploma in Mechanical Engineering and Aeronautics from the university of Patras (Greece) and a Master’s degree in Mechanical, Maritime and Materials Engineering department from TU Delft (the Netherlands). His research topics include the design and development of human-robot collaborative solutions, automation applications, cognitive mechatronics and PLC, Robot programming.

Dionisis Andronas, Senior Research Engineer and Project manager, LMS, University of Patras

Dionisis Andronas owns a master’s in mechanical and Aeronautics Engineering from the University of Patras. He is currently employed as a senior research engineer in the Laboratory for Manufacturing systems and Automation (LMS). He has been involved in a number of European Union funded projects, under the H2020 and FP7 programs, both as researcher and project manager. Through the years, he has elaborated on research topics dealing with robotics, human robot collaboration, workstation design, interfaces and design of mechatronics.

In this regard, large and fragile objects co-manipulation task has resulted in many research and development efforts in robotics field and in particular in the human-robot joint collaboration domain.

How to execute any trajectory in a human-robot collaborative mode while minimizing the stress applied to a fragile part by both partners, in order to avoid damaging it?

In the MERGING project, we propose a new collaborative robotic controller that fulfills the main requirements of co-transportation tasks of large and fragile parts, i.e. by executing any trajectory in a collaborative mode while minimizing the stress applied to the part by both partners in order to avoid damaging it. This is a necessary condition for such parts transportation. More especially, our controller prevents the robot from applying torques to the part, while maintaining a desired orientation of the part along the transport trajectory in order to follow the operator.

In the MERGING project, we designed a new robotic-assistant controller adapted to human-robot large and fragile parts transportation

The robot actively moves along the transport path according to its wrist’s angle. This angle results from the operator action of transporting the part. The transport path is described by a virtual guide that allows joint human-robot transportation. We implemented this virtual guide with the virtual mechanism (VM) principle [JOLY 1995] (1).
We defined this VM as a combination of a) MDSplines [SANCHEZ 2017] (2) to define the path trajectory, and b) of a spherical joint in order to prevent robot from applying torques to the part and to use the wrist’s angle to follow the operator along the path.

Our contribution is an extension of the assistance proposed in [SANCHEZ 2017] (2) and of the wrist’s angle use introduced in [HAYASHIBARA 1999] (3) in order to transport a large fragile part on any kind of trajectory and not only straight lines.

We recently led a pilot experiment that highlighted both the efficiency and ease of use of our approach for the transportation of a large and fragile part. On the one hand, our controller prevents torques application at both robot and operator gripping points, which minimizes stress on the part. On the other hand, the non-roboticists operators pointed out the high intuitiveness of our active assistance, in comparison to the passive assistance proposed in [SANCHEZ 2020] (2).

This work led to:
• A scientific paper  « Controller design of robotic assistant for the transport of large fragile part » presented at IROS 2022 International Conference
• A patent pending « Procédé de comanipulation d’une pièce par un opérateur aidé par un partenaire robotique »

(1) L. Joly and C. Andriot, “Imposing motion constraints to a force reflecting telerobot through real-time simulation of a virtual mechanism,” in IEEE International Conference on Robotics and Automation, vol. 1, 1995, pp. 357–362
(2) S. S´anchez Restrepo, G. Raiola, P. Chevalier, X. Lamy, and D. Sidobre, “Iterative virtual guides programming for human-robot comanipulation,” in IEEE International Conference on Advanced Intelligent Mechatronics (AIM), 2017, pp. 219–226
(3) Y. Hayashibara, T. Takubo, Y. Sonoda, H. Arai, and K. Tanie, “Assist system for carrying a long object with a human-analysis of a human cooperative behavior in the vertical direction,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, vol. 2, 1999, pp. 695–700

Julie Dumora
Julie DUMORA is a robotics research engineer at CEA. She received the PhD degree in 2014 in the field of human robot interaction (HRI). Her research focuses on comanipulation of large parts by a dyad composed of a human and a robot. Her activities aim at simplifying the deployment and use of robotics in plants. Her expertise focus mainly on control field for HRI and on intuitive programming for non-experts in robotics.

To be able to address the « Low volume / High mix » processes, a new approach is needed, that cuts down the programming time, and makes it accessible to non-expert people. The two main hurdles are generally the difficulty to program new behaviors, and to acquire the numerous points and trajectories needed for a given process.

Teaching robots points of interest and complex gestures is cumbersome and time consuming. How can we make it easier and faster?

In the MERGING project, CEA worked on this very topic, and developed a framework named SPIRE (Skill-based Programming Interface for Robotic Environments), that revolves around two core concepts: teaching by demonstration, and skill-based programming. This article gives an insight of this work, with a focus on the teaching part.

A skill is a high-level task-related robotic function, that allows to provide robotic actions in a meaningful way for the operator, with actions such as “insert pouch, place fabric, unwrap fabric”, and so on. The SPIRE skills framework is designed to provide a convenient and efficient way to program those skills, independently from the brand-specific API provided by the robot.

To be applied by the robot, a skill generally needs configuration, mainly from geometric information, such as points of application, gestures or trajectories to follow. In state of the art solutions, the teaching part is usually done using either raw data from CAD, the robot teach pendent, or hand-guiding features.

In the Merging project, we developed a new multimodal teaching module, which allows natural teaching using motion-capture and remote control. This MoCap-based approach is a great way to easily and quickly teach the robot. Motion-capture can be used “offline”, without the robot in the loop, to teach directly in the scene. It can also be used for robot teleoperation, enabling remote interaction of any kind of robots, even industrial robots in a dedicated cell. Each teaching modality has its own advantages and limitations, resumed in the following illustration. The choice will depend upon the situation.

By coupling motion-capture for teaching, and skills for robot capabilities programming, the SPIRE MERGING framework will enhance significally the user experience for collaborative robotics applications.

Baptiste Gradoussoff

Baptiste Gradoussoff is a research engineer and project manager at the Interactive Robotics Laboratory. He joined CEA LIST in 2011. Specialized on the topics of motion control and software for collaborative robotics and tele-manipulation, he steers the laboratory activities for agile robotics and intuitive programming, which goal is to develop efficient solutions for new stakes of manufacturing, such as small batch process automation

In MERGING we address the robotic, manipulation of complex objects and materials as the case of fabric manipulation to reduce wrinkles. The first step for the robotic manipulation of a fabric is the definition of the information required to perform the optimal actions for the manipulation of the material. For that purpose, the state of the system needs to be characterized, a prediction needs to be done to infer what is the next state of the system under the application of a given action, and a criterium to decide what action to take for a given target needs to be chosen.

What quantitative variables can be used in order to identify the state of a complex material and how to decide what to do in order to achieve a desired state?

Entropy is usually thought as a measurement of how much information a message or distribution holds. In other words, how predictable such distribution is. A fabric can present different surface related features (patterns, textures…) that do not hold information about the amount of wrinkles it has. Consider this fact, the definition of the state of a fabric has been addressed based on data experimentally acquired through pointclouds. In the context of MERGING, entropy gives an idea about the distribution of the orientations of the normal vectors of a given area of points related with the fabric observed. Based on this quantity, the state of the fabric can be defined, a target for this quantity can be stablished, and the actions needed to achieve such target can be addressed.

In MERGING a Reinforcement Learning framework has been defined in order to let an AI system explore about what are the consequences of fabric manipulations in terms of wrinkledness dynamics. Based on such observations, the system can infer what is the optimal action to apply on each situation to reduce wrinkles in a fabric based on real observations.

In order to address the massive complex manipulation of fabrics to reduce wrinkles over the surface a specific digital environment has been developed as it is the clothsim environment made available as open access as part of the results of the MERGING Project. The detailed description of the simulation can be found on its own repository.The clothsim environment has been used for the cloth initial random configuration and later the training of the system. The learning routine has suggested different actions considering the Q-values (Q-matrix in a Q-Learning context) by an argmax function. After the application of the actions clothsim returned the transition of the fabric and the values of the Q-matrix have been updated following a Q-learning update rule where a reward function stablishes how good or bad the action was attending to the calculated entropy for each state. The Q-values are updated including a reward function which becomes positive is the entropy is decreased and remains negative in other case.

In the real scenario, after one observation of the pointcloud of a fabric, in order to select the actions to apply, the knowledge of the system encoded in the classic Q-matrix which is inferred by the system for a given state can be exploited. Such codification is done using a matrix that consider corners to manipulate and directions that can be taken within the fabric manipulation. The final outcome of the procedure is a set of three points: one static point to fix the fabric, a second point that represents the corner to be manipulated and a third point which represents the point where this last corner has to be placed (grab point, pull point initial coordinates, pull point final coordinates).
The strategies have been also tested in a real scenario at AIMEN laboratory environment exploiting the knowledge gathered through training, the information captured through a point cloud and following the outcomes suggested by the analysis of the Q-matrix associated with the given state.

Our results show how entropy can be exploited as a metric to quantify the state of a complex material in order to manipulate it and drive a desired configuration. A training environment has been also presented as a requirement in order to deliver large amount of synthetic data that a Reinforcement Learning agent can acquire and later exploit in the real scenario.

Santiago Muiños Landin

Santiago Muiños Landin is the team leader in Artificial Intelligence and Data Analytics group at AIMEN. He has been involved in the definition of the Reinforcement Learning approach and contributed as the writer of this blog post.

Daniel Gordo Martín

Daniel Gordo Martín is an Industrial Engineer in Artificial Intelligence and Data Analytics at AIMEN. He has been an active contributing member of the MERGING project in validation and verification of the Artificial Intelligence approach and contributed as the writer of this blog post.

Acknowledgement: We would like to acknowledge the contribution of Carlos González Val, José Angel Segura Muros, David Castro Boga and Alberto Botana López for the Reinforcement Learning framework development, verification, and validation.

However, the intricate nature of the physical and mechanical characteristics of flexible materials presents a significant challenge when simulating and capturing their real-time behavior. The complexity arises from the fact that the deformation of an object is influenced not only by the external forces exerted upon it but also by its composition. Most of the time, the physical properties of these materials cannot be retrieved by already existing databases, especially when composite materials are concerned.

As a result, the full potential of planning and reasoning algorithms that rely on modeling techniques cannot be realized unless the object’s deformation is accurately represented. The calibration and fine-tuning of these models pose a significant challenge to industries dealing with pliable materials, hindering their advance towards automation and the Industrial metaverse.

 How do you bridge the gap between the real world and the simulated environment?

LMS has developed a novel methodology to seamlessly connect a deformable object’s digital representation with its physical counterpart. By deploying advanced optimization algorithms and artificial intelligence (AI) the proposed approach promises to enhance the accuracy and effectiveness of simulations in modeling and analyzing complex cyber-physical systems.

MERGING introduces a systematic procedure for the automated optimization of modeling parameters, combining accuracy and ease of integration for flexible materials.

Upon examination of the parameter optimization problem, it becomes apparent that two fundamental questions need to be answered:

1. How can we acquire a precise evaluation of the resemblance between the physical object and its model representation?
2. How can an efficient and effective approach be developed to navigate the extensive parameter space and discover optimal configurations for the non-linear and anisotropic equations that describe the deformable object’s behavior?

In our approach, to address these fundamental questions, the following strategies were adopted:

First and foremost, we need to gather data from reliable and credible sensors, ensuring accurate measurements of the object’s characteristics. This raw data is then processed and filtered to make it comparable with the reconstructed model. To facilitate comparison, the point-cloud structure proves to be highly advantageous. Employing point cloud comparison techniques, such as the chamfer distance metric, allows us to quantify the similarity between two point clouds and generate a “similarity score”.

To iterate through the vast amount of possible configurations for the model we need an efficient and reliable optimization module. Here, we present two distinct options within the module. The first option is Bayesian optimization, a renowned algorithm for globally optimizing black-box functions. Leveraging probabilistic modeling and intelligent exploration-exploitation trade-offs, Bayesian optimization facilitates efficient searching for the best solution amidst the parameter space. The second option is the CMA-ES algorithm, which introduces a novel approach to numerical optimization, harnessing the benefits of genetic algorithms. These two optimization options, integrated into the framework, provide flexible and powerful means to seek optimal configurations, significantly enhancing performance and efficacy in the parameter optimization process.

The experimental verification of the methodology’s results has been conducted by applying it to the reconstruction model of two-dimensional deformable objects, which was analyzed in a previous blog post. By capturing data from two distinct perception systems, an RGB-D sensor (ZED2 camera) and a laser-based sensor (LiDAR Velodyne puck), and deploying two optimization algorithms the methodology’s flexibility has been showcased.

The model’s parameter optimization has taken Merging’s digital twin to the next level. Promising to enhance and broaden the reach of robotic automation in non-rigid assemblies by providing reliable digital twins, the proposed methodology can become a valuable tool in cyber-physical systems and the industrial metaverse.

For more information you can refer to our publication:

Nikolaos Theodoropoulos, Emmanouil Kampourakis, Dionisis Andronas, Sotiris Makris, Cyber-physical systems in non-rigid assemblies: A methodology for the calibration of deformable object reconstruction models, Journal of Manufacturing Systems,
Volume 70, 2023, Pages 525-537

Nikolaos Theodoropoulos, Research Engineer, LMS, University of Patras.

Nikolaos Theodoropoulos works as a research engineer in the field of Robotics and Automation at the Laboratory for Manufacturing Systems and Automation (LMS). He holds a degree in Electrical and Computer Engineering from the University of Patras, where he established a solid foundation in computer science and electrical engineering. He has a background in software engineering, low-level programming, computer architecture, embedded systems design, and communication protocols. With a passion for technological advancements, Nikolaos’ research activities involve the development of control algorithms and hardware interfaces for robotic cells, the design and optimization of digital twins, and the development of computer vision algorithms for robotic applications.

In the current industrial landscape, flexible material manipulation predominantly involves a combination of manual efforts and semi-automated processes. Yet, these manual processes are labor-intensive, prone to inconsistencies, and lack scalability. Meanwhile, existing automated solutions often rely on rigid fixtures or simplified assumptions about the material properties, leading to limited flexibility and adaptability.

Moreover, the need for customizable handling interfaces for different robots, grippers, or software tools adds another layer of complexity. The ability to efficiently handle collisions and visualize complex fabric geometries in a simulation environment is vital for accurate and realistic modelling of deformable materials. So, how can we enhance the existing methods to address these challenges effectively?

What if we had an advanced tool that could accurately model and simulate
a broad range of fabric behaviors?

Introducing ClothSim, our game-changing tool engineered to simulate a wide array of fabric types under diverse conditions. By constructing a precise digital twin of the fabric, ClothSim enhances robot cognition of object deformation, paving the way for advanced manipulation of flexible materials.

MERGING introduces pioneering means for the reconstruction of two-dimensional deformable objects enabling fast, precise, and reliable fabric representation for robotic operations.

Building on the Provot’s mass-spring model, a staple in the computer graphics industry for several years, ClothSim pushes the boundaries by enhancing this model to capture the behavior of various fabrics with a high degree of accuracy. By integrating deformation constraints into the model, we prevent the over-extension of springs, thereby ensuring stable simulations that faithfully mirror the elastic properties of the fabric. The use of hard constraints further improves the model, allowing us to accurately simulate the interaction between the robot and the fabric.

To ensure the computational efficiency necessary for real-time applications, we leverage the power of Verlet integration. This numerical integration method strikes the perfect balance between accuracy and performance, allowing ClothSim to simulate fabric behavior in real-time. Our incorporation of non-linear springs enhances the versatility of ClothSim, making it capable of replicating a wide variety of fabric types.

Robustness is the cornerstone of ClothSim, with the introduction of dampers to mitigate the over- elasticity of the mass-spring model. By doing so, ClothSim ensures the stability of the model even under intensive and rapid manipulations.

At the heart of ClothSim is its adaptability. It’s thoughtfully designed to interface effectively with a wide array of robotic agents, grippers, and even virtual objects in the environment. Utilizing ROS integration, ClothSim delivers a modular framework that enhances adaptability and promotes efficient communication between different system components. This adaptability allows us to customize the interfaces to suit the needs of various applications and scenarios, establishing ClothSim as a versatile solution for a wide range of robotic applications.

ClothSim values realism. Our comprehensive collision handling approach, a blend of distance-based and penetration-based methods, ensures an authentic interaction between the fabric and the environment. Furthermore, ClothSim’s bitmapping methodology facilitates the accurate modelling and visualization of fabrics of complex shapes and geometries, expanding the range of applications that can benefit from our simulation tool.

ClothSim marks a substantial stride forward, bridging the gap between the virtual and real world, paving the way for enhanced robotic manipulation. With ClothSim, the scope of opportunities is expansive and diverse.

For more information you can refer to our publications:

S. Makris, E. Kampourakis, D. Andronas, “On deformable object handling: Model-based motion planning for human-robot co-manipulation”, CIRP Annals, Volume 71, Issue 1, pg. 29-32, (2022).

D. Andronas, E. Kampourakis, K. Bakopoulou, C. Gkournelos, P. Angelakis, S. Makris, “Model- Based Robot Control for Human-Robot Flexible Material Co-Manipulation”, 26th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), Available Online, 7-10 September, Västerås, Sweden (2022).

Emmanouil Kampourakis, Research Engineer, LMS, University of Patras.

Emmanouil Kampourakis works as a research engineer at the “Robotics” group of Laboratory for Manufacturing Systems and Automation (LMS). Bringing with him a solid foundation in software engineering, simulation models, control algorithms, embedded systems, and electronics, Emmanouil continually seeks to push the boundaries of knowledge in these fields. His research activities include the design of digital twins and simulation environments, cutting-edge algorithms for collaborative robotic cells and a plethora of interfaces utilizing state-of-the-art sensors and actuators.

How can a multi-modal perception system based on DL, using a few cameras for detection, tracking and gestures recognition, ensure safety without wearable?

MERGING perception system applies cameras and deep learning to detect and track workers without requiring them to wear any extra equipment. It deploys 2D and 360º cameras to create a multi-level perception system that recognizes single and multiple workers in a robot’s surroundings. The system also extracts information about workers’ gestures and limb positions for safety purposes. The perception system was validated at AIMEN laboratory environment.

The MERGING perception system detects, tracks, decomposes movements, and recognizes gestures of workers in a hybrid environment. It enables dynamic adaptation of robotic behavior for safe and natural human-robot interaction.

In this blog post, we summarize the assessment of a multi-level perception system that validates technologies for perception in human-robot interaction. The system involves non-wearable devices, such as person detection, tracking, movement decomposition, and gesture recognition for use in specific applications such as fabric detection, planning for wrinkle removal, pose extraction, continuous monitoring of fabric deformation, and continuous gripping monitoring. To achieve human detection and tracking, we deployed a calibrated stereo system consisting of two RGB cameras aided by YOLO filtering and pipelined the output for each detected person into OpenPose, which estimates the pixel position of 15 key points on the individual human body, mainly corresponding to body joints. Our implementation achieved a frame rate of 13 fps. The key point information was then fed into a gesture recognition algorithm that converted it into a line diagram of the human, focusing only on upper body key points. The gesture detection algorithm calculated the relative angle between the key points and returned a gesture by looking up the reference quaternion defined in a lookup table.
We verified and validated the system under actual industrial conditions before scheduling its deployment for the next few months. Simulated results show that the system is robust under industrial lighting conditions and can detect, track, and decompose human motion at high frame rates. However, we also highlight some limitations of the system, such as scenarios where two humans cross each other or wear clothing that matches the background color. In the future, improving the frame rate and addressing the system’s limitations will be a point of focus. For more details, please see the public deliverable D5.3 Perception functionalities demonstration you will find at https://www.merging-project.eu/news/public-deliverables/ .

The results show that our system can detect, track, and decompose movement of single or multiple humans with varied industrial clothing except for occlusion pose by crossing two humans and cloth-background color matching. In addition, our gesture recognition system was 97% successful for recognizing five human gestures.

Afra María Pertusa Llopis

Afra María Pertusa Llopis is a robotics engineer and a Ph.D. student in Advanced Robotic Technologies and Application at AIMEN. She has been an active contributing member of the MERGING project in validation and verification. She contributed as the writer of this blog

Jawad Masood

Jawad Masood is the team leader in Advanced Robotic Technologies and Application group at AIMEN. He has been involved in the application of sensors for human detection and tracking using DL approaches in automotive domain and wearable robotics. He contributed as the writer of this blog

Acknowledgement: We would like to acknowledge the contribution of Adriana Costas López and Diego Pérez Losada for the perception system development, verification, and validation.

Smaller product lifecycles require recurrent changes either on the bill of resources, their layout arrangement, or their program routines. Subsequently, there is a need for manufacturing lines to be flexible while also balancing quality, productivity and cost. A reconfigurable cell is almost equivalent to a cell with a lot of feasible arrangements of its elements. The number of possible execution flows, even for a static cell configuration, is big. In parallel, the sophistication of modern lines emerges challenges on how intelligent agents, multisensory systems and processing modules can be orchestrated.

Representative examples of sophisticated manufacturing systems are those involving deformable materials. Manipulating flexible materials like foams, cloths and wires requires complex handling strategies which frequently utilize flexible materials modelling frameworks, model-based planners, multi-sensory perception systems, and dexterous robot tools. This implies a large number of modules that need to be orchestrated for achieving the desirable performance, thus the time and resources for the creation, edit, maintenance, and monitoring of schedules can become a weak link for non-rigid assembly systems.

Designing an easy-to-use orchestrator with high parallelization capabilities requires striking an important balance

When designing a tool that orchestrates such complex robotic behaviors it is important to decide the way that execution flows are represented. That representation has to achieve an important balance. The created schedules need to be interpreted by the orchestrator quickly while being easy understood by users. Within MERGING, LMS designed the Deformable Object Handling Controller (DOHC) by combining common scheduling practices of manufacturing systems.

Schedules are described using a format that can visualize them using Activity on Node (AoN) diagrams. Those are easy for users and engineers to understand, and can be translated rapidly in Activity on Arc (AoA) diagrams, which can be interpreted and executed quickly by the orchestrator.

Additionally, a hierarchical approach is used, which makes it possible for users to describe complex execution flows without resulting in more complex representations for the orchestrator. Although this architecture doesn’t set any limits, for MERGING, five abstraction layers are used, namely: Order, Job, Task, Operation and Action.

Parallelization is achieved with the use of the actor model abstraction. Through a procedure that runs “under the hood”, without requiring any effort by the user, DOHC assigns activities to different actors and handles race conditions, making it possible for the described schedule to be executed in the fastest possible way. Whenever more parallel executions are possible, DOHC generates new actors to undertake them. DOHC can execute an amount of parallel running branches that is dependent on the hardware capabilities and is not limited by the software architecture.

It is evident, that even engineers with advanced programming skills may hesitate to use a tool that requires a tiresome learning process. Users that do not have programming skills have little to no chances of getting to know such technologies, especially when those are limited to a command line interface. Thus, an intuitive user interface is integrated in DOHC with the aim of enabling the use of the tool by users with less programming expertise. The interface allows for using DOHC via a graphical environment where schedules can be formed or maintained by simple “drag and drop” activities or by navigating through intuitive menus.

MERGING use cases constitute an opportunity to evaluate the developed software in state-of-the-art cells that require complex orchestration strategies. The results of early experimentations are promising, indicating that efficient runtime orchestration can be achieved with the use of DOHC. A significant improvement was also observed during the development phase, where it became possible to quickly create and test work schedules within simulation environment and easily overcome any identified problems.

Konstantinos Kavvathas, Research Engineer, LMS, University of Patras.

Konstantinos Kavvathas works as a research engineer at the “Robots, Automation and Virtual Reality in Manufacturing” group of Laboratory for Manufacturing Systems and Automation (LMS). He has background experience in machine learning, multidimensional data structures and web development. His research activities include the design of software for task planning and orchestration of reconfigurable or hybrid robotic cells through novel software applications.