Vascular cognitive impairment and dementia (VCID) which includes all forms of small vessel diseases is associated with white matter (WM) damages. Diffusion tensor imaging (DTI) has been used widely in several studies in characterizing these WM changes in VCID. The aim of this study is to evaluate the classification accuracy of diffusion measures to distinguish cerebral small vessel disease (CSVD) subjects from healthy controls (HC) and be further able to discriminate CSVD subgroups consisting of subcortical ischemic vascular disease , mixed dementia , leukoaraiosis , and Alzheimer’s disease. The proposed classification framework includes feature extraction followed by multiclass random forest classification with leave one out cross validation. The classification features were defined based on histogram measures calculated on 4 WM regions from four DTI modalities (FA, MD, AD, and RD). Multiple classification tasks were performed owing to the different subject groups. We have shown that DTI combined with FLAIR and cognitive score can give up to 78% classification accuracy for the 5-group and the 4-group classification without the need of the harder to measure cerebrospinal fluid biomarkers.