In simulations of colloidal matter, frictional contacts between particles are often neglected. For spherical colloids, such an approximation can be problematic, since frictional contacts couples translational and rotational degrees of freedom, affecting collective behavior of, e.g., colloids under shear and chiral active matter. While deterministic friction models exist in granular matter, colloidal systems require thermodynamically consistent treatment of thermal fluctuations. Here, we derive the correct fluctuation-dissipation relation for frictional contacts with Markovian, but otherwise arbitrary linear or nonlinear relation between the friction force and the relative velocity at the contact point, and how to implement the corresponding stochastic force terms. Among other, this creates a new generalized class of dissipative particle dynamics (DPD) thermostats with rotation-translation coupling. We demonstrate effects of frictional contact interactions using the example of Poiseuille flow and motility induced phase separation in active Langevin particles.