The physics of high harmonics has led to the generation of attosecond pulses and to trains of attosecond pulses. Measurements that confirm the pulse duration are all performed in the far field. All pulse duration measurements tacitly assume that both the beam's wavefront and intensity profile are independent of frequency. However, if one or both are frequency dependent, then the retrieved pulse duration depends on the location where the measurement is made. We measure that each harmonic is very close to a Gaussian, but we also find that both the intensity profile and the beam wavefront depend significantly on the harmonic order. Thus, our findings mean that the pulse duration will depend on where the pulse is observed. Measurement of spectrally resolved wavefronts along with temporal characterization at one single point in the beam would enable complete space-time reconstruction of attosecond pulses. Future attosecond science experiments need not be restricted to spatially averaged observables. Our approach paves the way to recovery of the single molecule response to the strong field.