We present a multi-site diffusion MRI (dMRI) data harmonization method using CycleGAN network with segmentation loss (CycleGANS). This method aims to learn an efficient mapping of dMRI signal using rotation invariant spherical harmonics features from the same set of subjects across sites. At the same time, it has potential to learn the tumor pathology (if exists) during harmonization. We compare our CycleGANS network with the CycleGAN network. We show that our CycleGANS network has better multi-site diffusion MRI data harmonization accuracy. Moreover, our method shows up to 60% improvement on the prediction of tumor pathology.