The original MD17 dataset (http://quantum-machine.org/datasets/#md-datasets) [Chemiela et al. Sci. Adv. 3(5), e1603015, 2017] contains numerical noise. Thus, any numbers presented from benchmarks on this data are likely flawed. Here, we present a new dataset with negligible numerical noise for benchmarking of forces and energy predictions for molecular dynamics simulations. As the structures are taken from a molecular dynamics simulation (i.e. time series data), they are not guaranteed to be independent samples. This is easily evident from the autocorrelation function for the original MD17 dataset. In short: DO NOT train a model on more than 1000 samples from the revised dataset, and do not train models for more than 50 samples from the original MD17 dataset. Data already published with 50K samples on the original MD17 dataset should be considered meaningless due to this fact and due to the noise in the original data.