Published this week in the Proceedings of the Royal B, researchers Kelley, Fitzpatrick, and others tested this question in their paper “Spots and stripes: ecology and colour pattern evolution in butterflyfishes.”
Butterflyfishes (Family: Chaetodontidae) were chosen for this study as it is a large family with a highly diverse coloration and pattern on their bodies with different genus’ having highly unique stripe, spot, and eyespots present.
The researchers used a dated molecular phylogeny of 95 species from this family (comprised about 75% of the total fish) and “tested whether spots and eyespots have evolved characteristics that are consistent with their proposed defensive function and whether the presence of spots and body stripes is linked with species’ body length, dietary complexity, habitat diversity or social behaviour.” For those unfamiliar with molecular phylogeny, it is “the analysis of hereditary molecular differences, mainly in DNA sequences, to gain information on an organism’s evolutionary relationships.” (Wikipedia)
Kelley’s group found stripe patters were consistent with different ecological factors including “habitat type, sociality and dietary complexity” supporting the supposition that stripe patters are evolutionarily important. However, eyespots and spots in general were found to have appeared relatively recently in butterflyfish, which casts doubt on the idea that spots and eyespots are used as a predator defense mechanism.