The evolutionary impact of structural variants, such as chromosomal inversions, is well documented, especially for their role in local adaptation in high gene flow systems. However, the role of other genomic rearrangements like chromosomal fusions, fissions, and translocations is still relatively unexplored. Here we present six chromosome-level Gadid reference genomes for the non-migratory Atlantic cod (Gadus morhua) i.e., Norwegian coastal cod (NCC), Atlantic haddock (Melanogrammus aeglefinus), burbot (Lota lota), European hake (Merluccius merluccius) as well as two keystone Arctic codfishes: the polar cod (Boreogadus saida) and Arctic cod (Arctogadus glacialis). Within a comparative genomics framework, we uncovered several lineage-specific chromosomal fusions, resulting in a reduced number of chromosomes compared to the ancestral state in the two cold-water adapted codfishes. The identified fusions were not homologous, i.e., indicating that they originate from independent evolutionary events. Additionally, a high number of partly overlapping chromosomal inversions between the two species were detected. Using a smaller population dataset, we uncovered a high degree of conservation for some of the overlapping inversions (including some breakpoint regions), suggesting that these regions are under selection, and potentially of evolutionary importance. With the use of chromosome-level genome assemblies, we demonstrate how large genomic reorganizations are likely to play important roles in speciation processes and thus, in particular to adaptation to freezing environmental conditions. Moreover, we observe that such massive rearrangement events can take place across relatively short evolutionary time scales.