We probe the accuracy limit of ab initio calculations of carrier mobilities in semiconductors, within the framework of the Boltzmann transport equation. By focusing on the paradigmatic case of silicon, we show that fully predictive calculations of electron and hole mobilities require many-body quasiparticle corrections to band structures and electron-phonon matrix elements, the inclusion of spin-orbit coupling, and an extremely fine sampling of inelastic scattering processes in momentum space. By considering all these factors we obtain excellent agreement with experiment, and we identify the band effective masses as the most critical parameters to achieve predictive accuracy. Our findings set a blueprint for future calculations of carrier mobilities, and pave the way to engineering transport properties in semiconductors by design.