Identifying knee osteoarthritis progressors is significant. MRIs can reflect the structures of the knee. However, currently no tool could rapidly and objectively predict knee osteoarthritis progression based on MRI. Therefore, we applied deep learning algorithms on MRIs of the whole knee to predict progression at three time points. The Gradient-weighted Class Activation Maps were employed for interpretability, and the highlighted infrapatellar fat pad (IPFP) was segmented for progression prediction. We showed that the deep learning framework performed well on discrimination of progressors, especially at 24th months, and that the infrapatellar fat pad plays an important role in predicting progression.