Using panel data from the Japanese Longitudinal Survey of Newborns in the 21st Century, the purpose of this paper is to investigate the factors that affect parents’ allocation of their time between work and child care. In relation to child care, we examine whether birth order and gender causally matter when parents make investments of their time in their children using a random effect instrumental variable estimator where a dummy for twins for the second birth is used as an instrument for family size. It is found that in Japan, daughters receive more time from their mothers, while fathers do not show any gender preference in their allocation of time across their children. There are important differences in this time allocation between working and non-working mothers, and the amount of time that is allocated to children on working and non-working days. Parents spend far more time with their first born child. These results are based on a sample of 74,526 working mothers, 62,709 non-working mothers, and 131,220 fathers. Using both random effect and fixed effects estimators, it is found as family size increases, the sexual division of labor in the household is deepened. That is, as the family size increases, the weekly working hours of fathers increase, while the weekly working hours of mothers decline. These results are based on a sample of 94,678 working mothers and 92,291 working fathers. In two-children families, the gender composition of the children does not seem to have any effect on the working hours of either parent. If the analysis is limited to those cases where the child surveyed is the first born, the gender of the child does not appear to have strong effects on the working hours of either mothers or fathers. Increases in the income of fathers leads to a reduction in the working hours of mothers, but an increase in the income of mothers leads to an increase in the working hours of fathers.