Models of visual cortex suggest that response selectivity can arise from recurrent networks operating at high gain. However, such networks have a number of problematic features: (i) they operate perilously close to a point of instability, (ii) small changes in synaptic strength can dramatically modify the degree of amplification, and (iii) they respond slowly to rapidly changing stimuli. Divisive inhibition, acting through interneurons that are themselves divisively inhibited, can solve these problems without degrading the selectivity of a recurrent network.