Savant syndrome is a phenomenon whereby individuals with cognitive impairments have one or more outstanding abilities, inconsistent with their general intellectual functioning. Approximately 50% of savant individuals have autism spectrum disorder (ASD), and 10–30% of people with ASD have savant skills. To shed additional light on this considerable overlap, we compared autistic traits as measured by the Social-Responsiveness-Scale (SRS) between 712 children with at least one reported savant skill, as determined by designated questions from the ADI-R questionnaire (savant group), and 2,032 non-savant children from the Simons-Simplex-Collection (SSC) database. We also examined SRS scores of the parents of these children and compared parent-child differences in SRS scores between the savant and non-savant groups. Savant children had significantly lower SRS scores (less deficiencies) compared to non-savant children (P < 0.05), while no such differences were observed among their parents. Further intra-familial analyses revealed weak pairwise-correlations (r = −0.015–0.141) between SRS scores of parents and their children, and significantly larger parent-child differences in standardized SRS scores within savant families (P < 0.05). These findings suggest that the less severe autistic traits among savant children with ASD compared to other people with ASD is not likely to be a familial trait.