Systematic comparison of tractography algorithms is rendered challenging by the differences in diffusion models and multiple parameters. In this work, we have defined a statistical framework to compare tractography algorithms, from two different perspectives: (1) the volume of generated tracts and (2) number of unbroken fibers. We applied our framework to compare six algorithms (four using tensor model and two using HARDI model) in reconstructing the contralateral corticospinal tract in ten brain tumor patients. Our results demonstrated significant differences between algorithms. These experiments establish our tractography comparison framework which can be used by researchers and tractography challenges for quantitative comparison.