There is growing interest in automating agricultural tasks that require intricate and precise interaction with specialty crops, such as trees and vines. However, developing robotic solutions for crop manipulation remains a difficult challenge due to complexities involved in modeling their deformable behavior. In this study, we present a framework for learning the deformation behavior of tree-like crops under contact interaction. Our proposed method involves encoding the state of a spring-damper modeled tree crop as a graph. This representation allows us to employ graph networks to learn both a forward model for predicting resulting deformations, and a contact policy for inferring actions to manipulate tree crops. We conduct a comprehensive set of experiments in a simulated environment and demonstrate generalizability of our method on previously unseen trees.
Tree structures can be represented as graphs to train the forward model and the contact policy.
Forward Model: Given an initial tree state and an applied action, the forward model predicts how the resulting tree has deformed by outputting per node differences.
Contact Policy: Given an initial tree state and target tree state, the contact policy predicts where and how to interact with the tree by outputting per node trajectory and per node affordance.
Our GNN model architecture utilizes stackable graph2graph layers, placed between input embedding layers and output prediction heads. The model takes as input a graph and outputs per-node predictions. Different output heads are used for the forward model and the contact policy
Initial tree (brown), contact node and trajectory (pink), predicted tree deformation (red), ground-truth tree deformation (green).
The forward model shown on varying tree structures and sizes. The forward model can generalize zero-shot to unseen tree sizes outside the training set.
Initial tree state (brown) and the desired target tree state (green). Given a set of candidate actions, we check feasibility and select the best action by iteratively applying RRT* to each candiate.
The contact policy used to manipulate the tree to a desired target state. The contact policy is shown on varying tree structures and sizes.