Several machine learning approaches have been used to classify brain tumors using MR images and spectra. Here we explore the specific properties of convolutional neural networks (CNN) for this task. We designed a CNN that could be trained on combined MR image and spectroscopic image data by exploiting their specific properties (spatial and spectral locality). Using a ‘leave-one-out’ validation, we demonstrate that our method outperforms state-of-the-art classification methods to distinguish tumor grades. These results demonstrate that CNNs are a powerful approach for tumor classification using MRSI data.