Cartilage thickness can be predictive of joint health. However, manual cartilage segmentation is tedious and prone to inter-reader variations. Automated segmentation using deep-learning is promising; yet, heterogeneity in network design and lack of dataset standardization has made it challenging to evaluate the efficacy of different methods. To address this issue, we organized a standardized, multi-institutional challenge for knee cartilage and meniscus segmentation. Results show that CNNs achieve similar performance independent of network architecture and training design and, given the high segmentation accuracy achieved by all models, only a weak correlation between segmentation accuracy metrics and cartilage thickness was observed.