The volatility implied by observed market prices as a function of the strike and time to maturity form an Implied Volatility Surface (IVS). Practical applications require reducing the dimension and characterize its dynamics through a small number of factors. Such dimension reduction is summarized by a Dynamic Semiparametric Factor Model (DSFM) that characterizes the IV S itself and their movements across time by a multivariate time series of factor loadings. This paper focuses on investigating long range dependence in the factor loadings series. Our result reveals that shocks to volatility persist for a very long time, affecting significantly stock prices. For appropriate representation of the series dynamics and the possibility of improved forecasting, we model the long memory in levels and absolute returns using the class of fractional integrated volatility models that provide flexible structure to capture the slow decaying autocorrelation function reasonably well.