We show that particle drift may play an important role in the transport of ultra-high energy cosmic rays (UHECRs) and their measured anisotropy, particularly when the transport is anisotropic. Drift and anisotropic diffusion has not been adequately included in previous studies. To fully account for the discreteness of UHECR sources in space and time, the Monte Carlo method is used to randomly place sources in the Galaxy and calculate the anisotropy of UHECR flux, given specific realisations of source distribution. We show that reduction in the rate of cross-field transport reduces the anisotropy. However, if the cross-field transport is very small, drift of UHECRs in the Galactic magnetic field (GMF) becomes the dominant contributor to the anisotropy. Test particle simulations further illustrate the effect of drift and verify our analytical calculation. The surprisingly low anisotropy measured by Auger can be interpreted as intermittency of UHECR sources, without invoking a flat source distribution and/or a high source rate.