Robust and Adaptive Dexterous Manipulation with Vision-Based Learning from Multi-Demonstrations

Abstract

Robotic manipulators perform tasks as humans with the potential to greatly assist people in industry, health care, and general society services. In this thesis, a vision-based learn from demonstration framework for a 7-degree-of-freedom robotic manipulator has been proposed. This framework enables learning from multiple human hand demonstrations to execute dexterous pick-and-place tasks. Conventional methods for collecting demonstration data involve manually and physically moving the robot. These methods can be cumbersome, lack dexterity, and be physically straining. The proposed contactless and markerless approach leverages MediaPipe software, dynamic time warping, Gaussian mixture model, and Gaussian mixture regression to capture and regress multiple dexterous hand motions. The proposed approach results in a more comprehensive motion representation, simplifying multiple demonstrations, and mitigating the non-smoothness inherent in a single demonstration. Dynamic movement primitives (DMP) with a force coupling term are employed to adaptively assimilate human actions into trajectories executable in dynamic environments. By considering the estimated variance from demonstration data, the path planning parameters are automatically fine-tuned and associated with the linear and nonlinear terms to adapt the trajectories. To compensate for unknown external disturbances, a non-singular terminal sliding mode controller (NTSMC) is applied to the Franka Emika robot for precise trajectory tracking. Experimental studies demonstrated the effectiveness and robustness of the proposed framework in executing human hand demonstrations, motion planning, and control for pick-and-place tasks.