We built an end-to-end deep learning model to rapidly stratify knees into morphological phenotypes using a large, longitudinal cohort with knee osteoarthritis (OA). We examined associations of phenotypes with odds of concurrent OA and OA progression. Bone, meniscus/cartilage, and inflammatory phenotypes were strongly associated with current structural OA and symptomatic OA. Hypertrophy phenotype was only weakly associated with structural OA. Among those who did not have baseline OA, bone and meniscus/cartilage phenotypes were strongly associated with developing both structural and symptomatic OA in 48 months. Only bone phenotype increased risk of undergoing total knee replacement surgery within 96 months.