Fiber orientation distribution function (fODF) is one of the key components for establishing brain connectivity maps. However, its reliable reconstruction usually requires a large number of diffusion weighted image (DWI) volumes leading to long acquisition time. Our previous study has shown the potential of multi-layer perceptron in recovering fODF directly from a small number of DWIs. In this study, we proposed a 3-dimentional convolution neural network to take the spatial correlation into consideration, allowing robust fODF reconstruction with up to eleven-fold reduction of number of DWIs. This method offers a new approach for fast fODF reconstruction which could facilitate its clinical applications.