1. The challenge of soft continuum robot control

Soft continuum robots driven by cables play a pivotal role in minimally invasive surgery (MIS) due to their potential to diminish lesions, pain, and the risk of infections^1,2. While the potential of using deep reinforcement learning (DRL) in controlling these robots has been explored³, significant discrepancies arise between simulations and real-world applications. Our objective is to bridge this gap by introducing a deep reinforcement learning method: the Axis-Space (AS) framework. This framework boosts computational speed and elevates control accuracy by minimizing sample complexity (SC) and curtailing the number of training steps.

2. Conceptual Foundation and Methodology

The primary conceptual foundation for our research was drawn from MIS, where surgical instruments access specific surgical targets within the human body via minor incisions. Herein, soft continuum robots demonstrate exceptional distal dexterity and structural flexibility, leading to their diverse designs tailored for MIS². Currently, controlling these robots remains a challenge due to the nonlinear attributes of the flexible manipulator, which encompasses structural deformations, interactions with malleable tissues, and potential collisions with other tools⁴. Existing value-based DRL algorithms, like the Deep Q-learning Network (DQN)³, have been employed for the 3-dimensional (3D) movement control of soft continuum robots. Although many scholars prefer DRL pretrained simulations to then be transferred to real-world settings^3,5,6, an evident disconnect persists between these two realms. Our solution is direct real-world robot training, eliminating the need for simulations. To achieve this, we put forth an intuitive method - the Axis-Space (AS) framework. This approach, emphasizing a redefined state and action space, leads to a substantial reduction in SC, making real-world robot training feasible and enhancing convergence rates.

3. Study Outcomes

Our research utilized a soft continuum robot equipped with four tendons per section as our experimental model. Controllers based on AS, namely ASDQN and ASDDQN, showcased superior performance in terms of reduced sample data needs and enhanced position tracking accuracy compared to their non-AS counterparts across various scenarios possibly encountered during MIS.

To assess controller models' capabilities in handling intricate tasks, we directed the continuum robot to trace two distinct reference trajectory patterns (sine function and square shape) using both ASDDQN and DDQN controllers. After completing each point tracking task, the robot reverted to its initial position. We also incorporated an inverse kinematics model-based optimal controller. Results depicted in Fig. 1a-b highlight that the ASDDQN controller, when used for these trajectories, brought down the RMSE values to 0.88 mm and 0.71 mm respectively. This marked an approximate 5-fold reduction from the IK model and nearly 50-fold from the DDQN controller. Such data emphasizes the ASDDQN controller's superior trajectory tracking capabilities compared to both its non-AS counterpart and IK controllers.

In the sphere of MIS, it's crucial for a soft continuum robot's tip to move with precision and robustness. Such attributes are indispensable for the robot to adeptly execute commands for non-linear trajectories, regardless of whether the environment is open or mimics the intricate structure of a biological body. Evaluating 3D trajectory tracking, we instructed the robot to follow diverse reference patterns, capturing the intricacies of 3D movement (Fig. 1c, circle, and Fig. 1d, 3D spiral shape). The tracking RMSEs for these patterns stood at 0.87 mm and 0.55 mm. To assess 3D point tracking, we directed the robot to trace distinct reference points on patterns, reflecting 3D movement complexities (Fig. 1e, Conical helix and Fig. 1f, Circle). The RMSEs for these patterns mirrored the earlier results. Additionally, for a 3D sine function pattern, the tracking RMSE was 0.65 mm (see Supplementary Figure 3). The findings reiterate that ASDDQN achieves submillimeter errors in tracking 3D shapes, underscoring its potential applicability in real surgical contexts.

Figure 1. Evaluation of different controllers under complex circumstances. (a, b) Sin (a) and square (b) function shape tracking under ASDDQN, DDQN and IK controllers. (c - f) Complicated pattern tracking under the ASDDQN controller, (c) Circle, (d) 3D spiral, (e) 3D flower, and (f) saddle surface. (SP: Start Position of robot; IK: Inverse Kinematic)

Reference

• Runciman, M., Darzi, A., & Mylonas, G. P. Soft Robotics in Minimally Invasive Surgery. Soft Robotics, 6(4), 423–443 (2019).
• Ji, G. L. et al. Towards Safe Control of Continuum Manipulator Using Shielded Multiagent Reinforcement Learning. IEEE Robot Autom Let 6, 7461-7468 (2021).
• Satheeshbabu, S., Uppalapati, N. K., Chowdhary, G. & Krishnan, G. Open Loop Position Control of Soft Continuum Arm Using Deep Reinforcement Learning. IEEE Int Conf Robot, 5133-5139 (2019).
• Webster, R. J., & Jones, B. A. Design and Kinematic Modeling of Constant Curvature Continuum Robots: A Review. The International Journal of Robotics Research, 29(13), 1661–1683 (2010).
• Miki, T. et al. Learning robust perceptive locomotion for quadrupedal robots in the wild. Sci Robot 7, eabk2822 (2022).
• Ding, Y. Lu, A. Kyme and S. S. Cheng, "Towards a Multi-imager Compatible Continuum Robot with Improved Dynamics Driven by Modular SMA," 2021 IEEE International Conference on Robotics and Automation (ICRA), 11930-11937 (2021).