Wheel Loader Scooping Controller Using Deep Reinforcement Learning

This article presents a deep reinforcement learning-based controller for an unmanned ground vehicle with a custom-built scooping mechanism. The robot's aim is to autonomously perform earth scooping cycles with three degrees of freedom: lift, tilt and the robot's velocity. While the majority of previous studies on automated scooping processes are based on data recorded by expert operators, we present a method to autonomously control a wheel loader to perform the scooping cycle using deep reinforcement learning methods without any user-provided demonstrations. The controller's learning approach is based on the actor-critic, Deep Deterministic Policy Gradient algorithm which we use to map online sensor data as input to continuously update the actuator commands. The training of the scooping policy network is done solely in a simplified simulation environment using a virtual physics engine, which converges to an average of a 65% fill factor from the full bucket capacity and a 5 [ sec] average cycle time. We illustrate the performance of the trained policy in simulations and in real-world experiments with 3 different inclination angles of the earth. An additional scooping experiment compared the performance of our controller to remote manual human control. Overall, the deep reinforcement learning-based controller exhibited good performance in terms of both achieved visually bucket fill with varying scooped earth weights of 4.1-7.2[kg], and a 5.1-7.1[sec] cycle time. The experimental results confirm the ability of our planner to fill bucket as required, indicating that our controller can be used for excavation purposes.