Motivated by the need for a positive-semidefinite estimator of multivariate realized covariance matrices, we model noisy and asynchronous ultra-high-frequency asset prices in a state-space framework with missing data. We then estimate the covariance matrix of the latent states through a Kalman smoother and expectation maximization (KEM) algorithm. Iterating between the two EM steps, we obtain a covariance matrix estimate which is robust to both asynchronicity and microstructure noise, and positive-semidefinite by construction. We show the performance of the KEM estimator using extensive Monte Carlo simulations that mimic the liquidity and market microstructure characteristics of the S&P 500 universe as well as in a high-dimensional application on US stocks. KEM provides very accurate covariance matrix estimates and significantly outperforms alternative approaches recently introduced in the literature.