Social interactions require fast and efficient person perception, which is best achieved through the process of categorization. However, this process can produce pernicious outcomes, particularly in the case of stigma. This study used fMRI to investigate the neural correlates involved in forming both explicit (“Do you like or dislike this person?”) and implicit (“Is this a male or female?”) judgments of people possessing well-established stigmatized conditions (obesity, facial piercings, transsexuality, and unattractiveness), as well as normal controls. Participants also made post-scan disgust ratings on all the faces that they viewed during imaging. These ratings were subsequently examined (modeled linearly) in a parametric analysis. Regions of interest that emerged include areas previously demonstrated to respond to aversive and disgust-inducing material (amygdala and insula), as well as regions strongly associated with inhibition and control (anterior cingulate and lateral prefrontal cortex). Further, greater differences in activation were observed in the implicit condition for both the amygdala and prefrontal cortical regions in response to the most negatively perceived faces. Specifically, as subcortical responses (e.g., amygdala) increased, cortical responses (e.g., lateral PFC and anterior cingulate) also increased, indicating the possibility of inhibitory processing. These findings help elucidate the neural underpinnings of stigma.