We use a long history of global temperature and atmospheric carbon dioxide (CO2) concentration to estimate the conditional joint evolution of temperature and CO2 at a millennial frequency. We document three basic facts. First, the temperature-CO2 dynamics are non-linear, so that large deviations in either temperature or CO2 concentrations take a long time to correct-on the scale of multiple millennia. Second, the joint dynamics of temperature and CO2 concentrations exhibit multimodality around historical turning points in temperature and concentration cycles, so that prior to the start of cooling periods, there is a noticeable probability that temperature and CO2 concentrations may continue to increase. Finally, evaluating the future evolution of temperature and CO2 concentration conditional on alternative scenarios realizing, we document that, even conditional on the net-zero 2050 scenario, there remains a significant risk of elevated temperatures for at least a further five millennia.