Automatic glioma grading based on magnetic resonance imaging (MRI) is crucial for appropriate clinical managements. Recently, Convolutional Neural Networks (CNNs)-based classification models have been extensively investigated. However, to achieve accurate glioma grading, tumor segmentation maps are typically required for these models to locate important regions. Delineating the tumor regions in 3D MR images is time-consuming and error-prone. Our target in this study is to develop a human knowledge guided CNN model for glioma grading without the reliance of tumor segmentation maps in clinical applications. Extensive experiments are conducted utilizing a public dataset and promising grading performance is achieved.