Spherical deconvolution for diffusion MRI requires a response function in order to accurately reconstruct the underlying voxel-wise fiber orientation distribution (FOD). Here, using 3D histologically-defined fiber orientation distributions and the corresponding diffusion signal, we derive the ground-truth fiber response functions. We show that there is significant variation in these response functions across the brain. We find that the current methods to estimate this function do not match the histological results, which leads to differences in fiber volume fractions. This is important because the wrong response function can amplify spurious peaks in the FOD and lead to inaccurate tractography.