Here we present data sets from direct numerical simulations of axisymmetric Taylor–Couette flow in the co-rotating regime that reveal a transition to spatio-temporal chaos as the system size increases. Beyond this transition, Taylor rolls suddenly undergo erratic drifts evolving on a very slow time scale. We estimate an effective diffusion coefficient for the drift and compare the dynamics to analogous motions in Rayleigh–Bénard convection and Poiseuille flow data sets, suggesting that this spontaneous diffusive displacement of large coherent structures is common among different types of wall-bounded turbulent flows.