In recent years, the birthrate has been continuously declining in Japan. The main causes of the decline are social factors. On the other hand, there is increasing evidence that many environmental chemicals show endocrine disrupting properties. Thus, we hypothesized that exposure to these chemicals would also be a causal for the fertility crisis. In this review, we examined current evidence that focused on environmental chemical exposure in utero and its association with reproductive hormones in children. We have included the findings from a prospective birth cohorts, the Hokkaido Study on Environment and Children’s Health Sapporo cohort. According to the literature, environmental chemical levels in utero, such as polychlorinated biphenyl, dioxins, perfluorinated chemical substances, phthalates, and bisphenol A were somewhat associated with the levels of reproductive hormones, such as testosterone, estradiol, progesterone, inhibin B, and insulin-like factor-3 in cord blood, in early childhood and adolescence. The literature also suggests the association between exposure to these chemicals and brain-sexual differentiation or the anogenital distance, which suggests the disruption of androgen shower during the developmental stage in the fetal period. There are still knowledge gaps on whether these hormones at an early stage affect the pubertal development and reproductive functions in later life. In addition, alternative chemicals are produced after banning one type. The health effects of alternative chemicals should be evaluated. Effects of exposure to a mixture of the chemicals should also be examined in future studies. In conclusion, the prevention of environmental chemical hazards in relation to human reproductive function is important. It would be one of the countermeasures to the falling birthrate caused by fertility issues.