In this abstract, we introduce a longitudinal diffusion-weighted infant brain atlas. For construction of this longitudinal atlas, we collected the diffusion-weighted images of 36 subjects, scanned at 5 time points (at neonate, 3, 6, 9 and 12 months of age). Our method builds the atlas from the diffusion-weighted images without the need for any diffusion models. Also, our method, based on patch-based sparse representation, preserves more structural details with spatial-temporal consistency in the longitudinal atlas. Thus, when applied to quantitative analysis of infant brain images, more accuracy can be achieved.