Adaptive radiation is characterized by the evolution of ecological diversity within a rapidly multiplying lineage. This process requires ecological opportunities to form new lineages, but also the evolutionary ability to realize this potential. In this light, the rapid diversification of some adaptive radiations is remarkable given the slow pace of mutation. The hamlets, a group of reef fishes from the wider Caribbean that radiated into a stunning diversity of color patterns, provide a compelling backdrop to investigate the genomic bases of diversification. Cross-coalescence analyses based on a dataset of 170 genomes representing 28 species pairs suggest that ancestral effective population size is high in the group (>10,000), and that radiation occurred within the last 10,000 generations. Seven percent of the samples were identified as high-probability hybrids or backcrosses, indicating that hybridization is ongoing at low levels. Genetic divergence (dXY) is largely driven by ancestral diversity, and genetic differentiation (FST) is restricted to a few narrow genomic intervals. These genomic regions are associated with specific components of color pattern (bars, marks and color) that form the basis of phenotypic variation in the group. Together our data reveal that the hamlet radiation is fueled by ancestral variation, hybridization, and a modular genetic basis of color pattern whereby phenotypic diversity is generated by different combinations of alleles at a few large-effect loci. Such a modular genetic basis of diversification has been reported in birds and butterflies and may underlie a variety of adaptive radiations on land, in freshwater and in the sea.