Primate brain evolution shows progressive enlargement, yet some lineages exhibit striking reductions in relative neocortex size. These cases remain poorly understood but can illuminate the constraints shaping neural evolution. Using phylogenetic modeling, we investigated neocortex evolution in New World monkeys and found the strongest reduction in marmosets, tamarins, and owl monkeys—small primates with rapid life histories, consistent with evolutionary miniaturization. To investigate the genomic correlates of relative neocortex reduction, we scanned >11,000 protein-coding orthologous genes using selection tests. aBSREL identified 47 genes under positive selection in the lineage, in neurodevelopment and growth regulation (e.g. LHX3, EPHB2, PDE2A). Complementary analyses using RELAX tests revealed 67 genes with intensified selection, enriched for brain expression, including DISC1 (neuronal migration), LAMC3 (cortical patterning), NEPRO (neural progenitor regulation), and RTTN (microcephaly). These findings reveal complex primate brain evolutionary trajectories. Molecular signatures indicate neocortical shrinkage was not a passive consequence of body size reduction, but an active evolutionary process with potential cognitive implications. Detection of pronounced neocortical reduction in marmosets—an important neuroscience model—provides new context for interpreting their neural organization.