Quantifying and modelling decay in forecast proficiency indicates the limits of transferability in land-cover classification

The ability to provide reliable projections for the current and future distribution of land-cover is fundamental if we wish to protect and manage our diminishing natural resources. Two inter-related revolutions make map productions feasible at unprecedented resolutions—the availability of high-resolution remotely sensed data and the development of machine-learning algorithms. However, ground-truthed data needed for training models is in most cases spatially and temporally clustered. Therefore, map production requires extrapolation of models from one place to another and the uncertainty cost of such extrapolation is rarely explored. In other words, the focus has mainly been on projections, and less on quantifying their reliability. Using the concept of “forecast horizon”, we suggest that the predictability of land-cover classification models should be quantitatively explored as a continuum against distances measured along multiple dimensions—space, time, environmental and spectral. Focusing on ten agricultural sites from England and using models specifically designed to predict multivariate decay-curves, we ask: how does a model's predictive performance decay with distance? More specifically, we explored if we could predict the proficiency (kappa statistics) of a model trained in one site when making predictions in another site based on the spatial, temporal, spectral and environmental distances between sites. We found that model proficiency decays with distance between sites in each dimension. More importantly, we found for the first time, that it is possible to predict the performance a model transferred to or from a novel site will have, based on its distances from known sites. The spectral distance variables were the most important when predicting model transferability. Exploring model transferability as a continuum may have multiple usages including predicting uncertainty values in space and time, prioritisation of strategies for ground-truth data collection, and optimising model characteristics for defined tasks.