Economists disagree about the factors driving the substantial increase in residual wage inequality in the U.S. over the past few decades. We identify and estimate a general model of log wage residuals that incorporates: (i) changing returns to unobserved skills, (ii) a changing distribution of unobserved skills, and (iii) changing volatility in wages due to factors unrelated to skills. Using data from the Panel Study of Income Dynamics, we estimate that the returns to unobserved skills have declined by as much as 50% since the mid-1980s despite a sizable increase in residual inequality. Instead, the variance of skills rose over this period due to increasing variability in life cycle skill growth. Finally, we develop an assignment model of the labor market and show that both demand and supply factors contributed to the downward trend in the returns to skills over time, with demand factors dominating for non-college-educated men.