Here we present data sets from direct numerical simulations of axisymmetric Taylor--Couette flow in the co-rotating regime that reveal a transition tospatio-temporal chaos as the system size increases. Beyond this transition, Taylor rolls suddenly undergo erratic drifts evolving on a very slow time scale.We estimate an effective diffusion coefficient for the drift and compare the dynamics to analogous motions in Rayleigh--Bénard convection and Poiseuilleflow data sets, suggesting that this spontaneous diffusive displacement of large coherent structures is common among different types of wall-boundedturbulent flows.