The correlation between historical and realized volatilities is studied empirically for a large range of time intervals. Similarly, the correlation between the volatility changes and the realized volatilities is studied. Both quantities measure the response functions of the market participants. These correlations show explicitly the heterogeneous structure of the market according to the characteristic time horizons of the different agents. It reveals a volatility cascade from long to short time horizons, with a structure different from the one observed in turbulence. A comparison is made with several theoretical processes used in finance, allowing a better understanding of the role and interactions of the market participants (intra-day trader, portfolio manager, central banks, pension funds, …). Moreover, we have developed a new ARCH-type process that incorporates the different groups of agents, with their characteristic memories. This process reproduces well the empirical response function, and allows us to quantify the importance of each group.