Contrast-enhanced MR plays an important role in the characterization of hepatocellular carcinoma (HCC). In this work, we propose an attention-based common and specific features fusion network (ACSF-net) for grading HCC with Contrast-enhanced MR. Specifically, we introduce the correlated and individual components analysis to extract the common and specific features of Contrast-enhanced MR. Moreover, we propose an attention-based fusion module to adaptively fuse the common and specific features for better grading. Experimental results demonstrate that the proposed ACSF-net outperforms previously reported multimodality fusion methods for grading HCC. In addition, the weighting coefficient may have great potential for clinical interpretation.