The article analyses how living conditions during childhood and adolescence structure socio-economic circumstances in midlife. The data are drawn from a new longitudinal Swedish data set—the Stockholm Birth Cohort Study—in which we can follow 14,294 individuals from birth (1953) to the age of 48 (2001). The analysis proceeds in three steps. The first step establishes the link between precarious living conditions in childhood and midlife social exclusion. In the second step, structural equation modelling is used to depict the pathways by which this association is mediated. The analysis produces tentative evidence that the long-term effect of financial poverty primarily runs via educational failure, whereas the effect of other social problems in the family of origin runs via deviant behaviour. In the third step, we analyse whether or not children who were raised in poor families or in families with other social problems are more sensitive to new risk exposures as adults. This is tested by examining the effect of long-term unemployment during the economic crisis of the 1990s on social exclusion risks 7–9 years later.