Construction of brain atlases is generally carried out using a two-step procedure involving registering a population of images to a common space and then fusing the aligned images to form an atlas. In practice, image registration is not perfect and simple averaging of the images will blur structures and cause artifacts. In diffusion MRI, the problem is even more challenging, since the alignment of gross anatomical structures does not necessarily guarantee the alignment of the microstructural information captured in each voxel. In this situation, it is unclear for example how signals characterizing fiber bundles of varying orientations, which can occur naturally across subjects, should be fused to form the atlas. Moreover, the commonly used simple averaging method is sensitive to outliers.