There is increasing experimental and neuropsychological evidence that action selection is directly constrained by perceptual information from objects as well as by more abstract semantic knowledge. To capture this evidence, we develop a new connectionist model of action and name selection from objects - NAM (Naming and Action Model), based on the idea that action selection is determined by convergent input from both visual structural descriptions and abstract semantic knowledge. We show that NAM is able to simulate evidence for a direct route to action selection from both normal subjects (Experiments 1 and 2) and neuropsychological patients (Experiments 3-6). The model provides a useful framework for understanding how perceptual knowledge influences action selection.