Shape analysis, a field in need of careful validation

In the last two decades, the statistical analysis of shape has become an actively studied field and finds applications in a wide range of areas. In addition to algorithmic development, many researchers have distributed end-user orientated toolboxes, which further enable the utilization of the algorithms in an "off the shelf" fashion. However, there is little work on the evaluation and validation of these techniques, which poses a rather serious challenge when interpreting their results. To address this lack of validation, we design a validation framework and then use it to test some of the most widely used toolboxes. Our initial results show inconsistencies and disagreement among four different methods. We believe this type of analysis to be critical not only for the community of algorithm designers but also perhaps more importantly to researchers who use these tools without knowing the algorithm details and seek objective criteria for tool selection.