X-ray photons, because of their long mean-free paths, can easily escape the galactic environments where they are produced, and interact at long distances with the intergalactic medium, potentially having a significant contribution to the heating and reionization of the early universe. The two most important sources of X-ray photons in the universe are active galactic nuclei (AGNs) and X-ray binaries (XRBs). In this Letter we use results from detailed, large scale population synthesis simulations to study the energy feedback of XRBs, from the first galaxies (z~20) until today. We estimate that X-ray emission from XRBs dominates over AGN at z>~6-8. The shape of the spectral energy distribution of the emission from XRBs shows little change with redshift, in contrast to its normalization which evolves by ~4 orders of magnitude, primarily due to the evolution of the cosmic star-formation rate. However, the metallicity and the mean stellar age of a given XRB population affect significantly its X-ray output. Specifically, the X-ray luminosity from high-mass XRBs per unit of star-formation rate varies an order of magnitude going from solar metallicity to less than 10% solar, and the X-ray luminosity from low-mass XRBs per unit of stellar mass peaks at an age of ~300Myr and then decreases gradually at later times, showing little variation for mean stellar ages >~3Gyr. Finally, we provide analytical and tabulated prescriptions for the energy output of XRBs, that can be directly incorporated in cosmological simulations.