This study investigated the value of deep learning for the detection and differential diagnosis of vertebral fracture. A model using ResNet50 was developed and tested in a separate dataset. The results were compared with the interpretation of an experienced radiologist. Our study noted that the analysis based on single vertebral body without inclusion of the soft tissue, the posterior elements, and the skipped lesions might be the reason why the radiologist’s reading was better than deep learning approach. For the identification of malignant fracture using whole images from training set, the prediction accuracy was only moderate, with rooms for improvement.