ESR12 - Grasping and Manipulation

Implementation of spiking tactile system on a 7-DoF manipulator and development of manipulation algorithms for stable grasping of objects of different shape and softness.

Overview

Objectives

Implementation of spiking tactile system on a 7-DoF manipulator and development of manipulation algorithms for stable grasping of objects of different shape and softness. Tactile data will be integrated in the control loop of the manipulator to obtain optimal grasps taking the object shape, softness and deformability into account. Different types of end-effectors will be considered, i.e. from parallel grippers to under-actuated multi-fingered hands.

Expected Results

A new set of force controllers, i.e. impedance and admittance controllers, will be obtained allowing different robots, from mobile manipulators to humanoid robots, performing grasping of a large variety of objects. The performance of the new grasping algorithms will be compared to the performance obtained using grasping algorithms based on open-loop approaches and those relying on data provided by a wrist-mounted force/torque sensor. The goal is to show that the new algorithms based on the tactile system provide a much higher ratio of successful grasps considering a set of objects and different configurations. The demonstrator will feature a service robot being able to grasp a variety of objects involved in typical home daily tasks of elderly care scenarios

Secondments

IIT-iCub

implementation of spiking tactile skin for robots
USFD

learn predictive methods
UNIBI

learn manipulation of switches

Supervisors

S. Terreri
C. Bartolozzi
L. Natale
T. Prescott
R. Haschke

Luca Lach

My interest in Robotics and Cognitive Science was sparked when I joined the RoboCup @Home Team TOBI. In 2017, we won the WorldChampionship in service robotics and for my Bachelor's Thesis I've work with one of these robots. During that time, I learned how robots drive, manipulate objects, form decision processes and how they can learn themselves. For my Master's Thesis, I explored the field of Reinforcement Learning in conjunction with Representation Learning. It was very exciting for me to acquire new skills in this field of research. My plan is to combine both of my main interest for my PhD.

Neutouch for me

What I like most about NeuTouch is the mix of many different sciences which try to achieve a common goal. The possibility to experience a different culture for three years motivated my choice as well. I hope that this time will create many interesting opportunities for exciting collaborations.

Info

Research Topics

Robotics, machine learning
Institution

PAL robotics
Background

B.Sc. Cognitive Computer Science (2013-2017)
M.Sc. Intelligent Systems (2017-2019)

Thesis

Safe object manipulation strategies using tactile sensors. University of Bielefeld, 2024.

Abstract

Tactile feedback is essential for object manipulation tasks for humans and robots, facilitating reactive control even without vision. This dissertation introduces several tactile-based methodologies to enhance the manipulation performance and safety of robotic systems. Before presenting these methods, the dissertation offers a high-level overview of tactile feedback in human manipulation tasks. Research indicates that humans divide manipulation tasks into distinct action phases, each defined by specific subgoals and guided by tactile cues. This concept is further refined in this thesis through the introduction of three manipulation task stages: grasping, manipulation, and placing. These stages provide a structured framework for categorizing the proposed methods and organization of the thesis. The first approach introduced in this thesis is a novel grasp force controller that is split into control phases, similar to human grasping, and implements several object safety features: It prevents undesired object motions during the grasp by halting finger movements based on tactile cues. Additionally, it maintains a given target force even under external disturbances while holding the object. Subsequently, a reinforcement learning scheme is introduced that enables zero-shot sim-to-real transfer of continuous grasp force control policies. A novel simulation environment is proposed to facilitate this, modeling a grasping scenario with realistically varied objects in size and stiffness. A comparison of these two methods facilitates a broader discussion on the advantages of learning-based methods while also highlighting the beneficial properties of classical controllers. For the manipulation stage, a simulation-based study is detailed, proposing and comparing different tactile sensor configurations for an anthropomorphic robot hand. The performance of each sensor configuration is evaluated by integrating its data into a policy trained on various in-hand manipulation tasks. This comparison of experimental results leads to the formulation of recommendations for developing future end-effector sensorizations. In the fourth and final approach, a supervised learning method for object placing is proposed. Given the challenges of occlusions in vision-based methods in placing scenarios, tactile data proves particularly valuable. The thesis concludes with a comparative analysis of the models, tasks, and sensors employed across the presented approaches.

Publications

Lach, L., Haschke, R., Tateo, D., Peters, J., Ritter, H., Borràs, J., & Torras, C. (2024, October). Zero-Shot Transfer of a Tactile-based Continuous Force Control Policy from Simulation to Robot. In 2024 IEE

Lach, L., Haschke, R., Tateo, D., Peters, J., Ritter, H., Borràs, J., & Torras, C. (2023). Towards transferring tactile-based continuous force control policies from simulation to robot. arXiv preprint arXiv:2311.07245.

Lach, L., Lemaignan, S., Ferro, F., Ritter, H., & Haschke, R. (2022, October). Bio-inspired grasping controller for sensorized 2-dof grippers. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 11231-11237). IEEE.

Melnik, A., Lach, L., Plappert, M., Korthals, T., Haschke, R., & Ritter, H. (2021). Using tactile sensing to improve the sample efficiency and performance of deep deterministic policy gradients for simulated in-hand manipulation tasks. Frontiers in Robotics and AI, 8, 538773.

Melnik, A., Lach, L., Plappert, M., Korthals, T., Haschke, R., & Ritter, H. (2019, November). Tactile sensing and deep reinforcement learning for in-hand manipulation tasks. In IROS workshop on autonomous object manipulation (Vol. 39, pp. 3-20).

Lach, L., Funk, N., Haschke, R., Lemaignan, S., Ritter, H. J., Peters, J., & Chalvatzaki, G. (2023, October). Placing by Touching: An empirical study on the importance of tactile sensing for precise object placing. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 8964-8971). IEEE.

Lach, L., Haschke, R., & Ferro, F. (2020). Leveraging Touch Sensors to Improve Mobile Manipulation. arXiv preprint arXiv:2010.10810.

Lach, L., Haschke, R., & Ferro, F. (2021). Improving Manipulation Performance of Mobile Robots Using Tactile Sensors. In 10th International IEEE EMBS Conference on Neural Engineering (NER’21).

Niemann, C., Leins, D., Lach, L., & Haschke, R. (2024, October). Learning When to Stop: Efficient Active Tactile Perception with Deep Reinforcement Learning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 685-692). IEEE.

Lach, L., Ferro, F., & Haschke, R. (2023). TIAGo RL: Simulated Reinforcement Learning Environments with Tactile Data for Mobile Robots. arXiv preprint arXiv:2311.07260.

Top