It is commonly believed that the correlations between stock returns increase in high volatility periods. We investigate how much of these correlations can be explained within a simple non-Gaussian one-factor description with time-independent correlations. Using surrogate data with the true market return as the dominant factor, we show that most of these correlations, measured by a variety of different indicators, can be accounted for. In particular, this one-factor model can explain the level and asymmetry of empirical exceedance correlations. However, more subtle effects require an extension of the one-factor model, where the variance and skewness of the residuals also depend on the market return.