Most literature on gender and mathematics is based on assumptions that the value of mathematical accomplishment is unquestionable and wholly good, and that any failures of women to achieve in mathematics are due to problems within her or the larger society, problems which should and can be resolved. This article argues that the situation is not so simple. Drawing on literatures of sociology, cultural studies, and education (among others), the article sets forth characteristics shared by 'marked categories' of persons as deviant. Drawing evidence from a wide range of sources, it is then argued that 'the mathematically able' share these characteristics and can therefore be considered as a marked category. Moreover, women in mathematics are doubly marked (as women and as mathematical), making their position doubly difficult socially. There are thus two discourses of mathematics in Western society, a discourse of power and a discourse of deviance. Drawing on the practices of other marked categories, I make a few recommendations for educators based on the analysis.