The isostructural family of compounds that includes the familiar copper sulfate pentahydrate and the two less familiar title materials, has for decades been a model for studying a range of structural and magnetic phenomena. Our understanding of the magnetic transitions on cooling below 1 K have hitherto been based on an assumption that there is no preceding structural change, an assumption we now know to be false. We have shown that CuSO4.5D2O undergoes a loss of inversion symmetry at 32 K, with an associated cell doubling along the a- and c-axis directions. Curiously, there is some evidence in the literature for such a transition in the fluoroberyllate analogue but not the selenate or sulfate. Our aim here is to determine if the purported structural transition in the BeF4 analogue is the same as we observe in the sulfate and to test whether it also occurs in the selenate.