Automated segmentation using deep learning can potentially expedite segmentation tasks. However, the generalizability of such algorithms on new unseen datasets is unknown. To test this generalizability, we used a knee segmentation algorithm trained on Osteoarthritis Initiative double-echo steady-state (DESS) datasets to segment cartilage from quantitative DESS datasets from three independent studies. We compared manual-automatic segmentation accuracy and the resultant qDESS T2 map variations. These results quantitatively demonstrate that a deep learning network trained on a single dataset does not generalize with a high accuracy to additional datasets even with similar image characteristics, and that additional fine-tuning may be needed.