Coral reefs rank among the most diverse species assemblages on Earth. A particularly striking aspect of coral reef communities is the variety of color patterns displayed by reef fishes. Color pattern is known to play a central role in the ecology and evolution of reef fishes through e.g. signaling or camouflage. Nevertheless, color pattern is a complex trait in reef fishes---actually a collection of traits---that is difficult to analyze in a quantitative, standardized and automated way. This is the challenge that we address in this study using the hamlets (\textit{Hypoplectrus} spp, Serranidae) as a model system. Our approach involves a custom underwater camera system to take orientation- and size-standardized photographs \textit{in situ}, color correction, alignment of the fish images with a combination of landmarks and Bézier curves, and Principal Component Analysis (PCA) on the color value of each pixel of each aligned fish. We complement this quantitative and standardized analysis of color pattern with whole-genome sequencing to run a multivariate Genome-Wide Association Study (GWAS) for color pattern variation. This approach reveals sharp association peaks along the hamlet genome and allows to characterize the phenotypic effect of the Single Nucleotide Polymorphisms (SNPs) that are most strongly associated with color pattern variation at each association peak. Major color pattern elements such as vertical bars emerge from this analysis, as well as a number of more subtle elements. Our results suggest that the diversity of color patterns displayed by the hamlets is generated by a modular genomic and phenotypic architecture.