Mind the Gap: Delayed Label Bias-Variance Tradeoffs in Predicting Likelihood of Nonpayment

The purpose of an online electronic-payment risk detection system is to prevent leakage, i.e., the loss of revenue that occurs when users fail to pay for services or when transactions are reversed. Nonpayment prediction models are trained on datasets comprising of features available when the model is triggered and the corresponding nonpayment labels. The latter are typically only observed several weeks or even months later. Furthermore, behavior indicative of future nonpayment is highly non-stationary, and the true model may drift significantly in the gap between trigger events and label collection. To address these challenges, we use post-transaction signals to generate pseudo-labels, i.e., short-term proxies [23] or surrogate-indices [33]. Our framework attains a favorable tradeoff between ameliorating bias due to drift and introducing variance due to pseudo-label noise, as demonstrated by both offline and online experiments on several nonpayment-detection systems at Meta. Our deployment on live user traffic yields a statistically significant improvement in revenue, accounting also for leakage.