Perceptual resolutions can be improved by contextual influences. This is exemplified in the tendency to be biased towards the center of stimulus sample (regression-to-the-mean (RTM)), thus optimizing outcomes by accounting for noise. We examined whether contextual influences may underlie refined face representations shaped by experience, and the reduced perceptual resolution exhibited in less frequently encountered faces as evident in the “other-race effect”. Participants performed a same/different task on two consecutive faces drawn from a sample of 100 morph levels created from two faces. Order of presentation was manipulated in different trials: in half of the trials the first face was closer to the mean of the morph continuum (bias+), and in the other half, the first face was further away from the mean (bias-). We surmised that RTM would be manifested in better performance in the bias+ trials, where perceived distance between the two faces increases because of regression of the first face towards the mean. Experiment 1 revealed that for own-race faces, all participants exhibited RTM evident during the first or second session. The extent of the RTM effect correlated with performance on face perception tests, indicating its relevance to face processing. Experiment 2 revealed that for other-race faces, RTM was evident in all participants already during the first session. However, the other-race effect was demonstrated for the bias- trials, but was diminished for the bias+ trials, implying that the poor face representations of other-race faces may originate from a weak reference. When a strong reference facilitated discrimination (bias+ trials), the disadvantage of other-race faces was no longer evident. The results suggest that better resolution of face representation can be achieved when a strong reference of a typical, averaged face is established. Moreover, mechanisms of perceptual inference enhancing general perceptual resolutions may underlie the way experience refines face representations.