This paper reviews research examining the role of visual memory in scene perception and visual search. Recent theories in these literatures have held that coherent object representations in visual memory are fleeting, disintegrating upon the withdrawal of attention from an object. I discuss evidence demonstrating that, far from being transient, visual memory supports the accumulation of information from scores of individual objects in scenes, utilizing both visual short-term memory and visual long-term memory. In addition, I review evidence that memory for the spatial layout of a scene and memory for specific object positions can efficiently guide search within natural scenes.