This paper presents a formal syntax and semantics for computation in neural networks. The main motivation for this is to provide a foundation for rigorous mathematical analysis of the capabilities of neural networks in relation to other types of computational system. A secondary benefit is that it helps to clarify obscurities and controversial issues in the notion of connectionist computation as it is informally understood. The paper reviews the various informal and formal definitions of connectionism in the literature and attempts to identify common principles and areas of disagreement. Central to connectionism is the idea of a system of simple nodes working together to solve a task, where each node acts in a purely local way on its neighbours: the vague words 'simple' and 'local' are clarified and defined by my formal system in a precise way, free from arbitrary restrictions. The system also defines the semantics of node growth, node pruning and connectivity change-operations that are used in an increasing number of recent connectionist algorithms but are not taken into account by previous definitions of connectionism.