Rapid build-up of greenhouse gases is expected to increase Earth' s mean surface temperature, with unclear effects on temperature variability. This makes understanding the direct effects of a changing climate on human health more urgent. However, the effects of prolonged exposures to variable temperatures, which are important for understanding the public health burden, are unclear. Here we demonstrate that long-term survival was significantly associated with both seasonal mean values and standard deviations of temperature among the Medicare population (aged 65+) in New England, and break that down into long-term contrasts between ZIP codes and annual anomalies. A rise in summer mean temperature of 1 °C was associated with a 1.0% higher death rate, whereas an increase in winter mean temperature corresponded to a 0.6% decrease in mortality. Increases in standard deviations of temperature for both summer and winter were harmful. The increased mortality in warmer summers was entirely due to anomalies, whereas it was long-term average differences in the standard deviation of summer temperatures across ZIP codes that drove the increased risk. For future climate scenarios, seasonal mean temperatures may in part account for the public health burden, but the excess public health risk of climate change may also stem from changes of within-season temperature variability.