Segmentation and motion parameter estimation for robotic Medjoul-date thinning

Laborious fruit thinning is required for attaining high-quality Medjoul dates. Thinning automation can significantly reduce labor and improve efficiency. An image processing apparatus developed for robotic Medjoul thinning is presented. Instance segmentation based on Mask R-CNN was applied to identify the fruit bunch components: spikelets and rachis. Motion planning parameters were extracted using the derived masks: rachis center point (RCP), rachis orientation angle, and spikelets remaining length. RCP and rachis orientation angle were computed geometrically, spikelets remaining length was estimated with a convolutional neural network (CNN) and a deep neural network (DNN). Instance segmentation results were accurate, especially for spikelets, for low intersection over union (IoU) (0.3 IoU, fruit determined for thinning identification, spikelets: 98%, rachises: 73%). However, only 66% of the rachises were correctly matched to spikelets. The segmentation of all spikelets and rachises in the images was of medium quality for low IoU (0.3 IoU, F1, spikelets: 0.67, rachis: 0.77), where both precision and recall dropped for higher IoUs. RCP and rachis orientation angle were accurately estimated (0.3 IoU, error, RCP: 2.2 cm, rachis orientation angle: 5.0°). Spikelets remaining length estimation using CNN resulted in better performance than DNN (0.3 IoU, error, CNN: 19.7%, DNN: 24.6%). Spikelets segmentation results are suitable for thinning automation. However, rachis segmentation and matching the rachis and spikelets may still require human intervention during run-time. RCP and rachis orientation angle estimation errors are acceptable, while spikelets remaining length estimation errors are acceptable only for preliminary motion planning and mandate additional tuning during motion execution.