We propose a novel task based deep learning framework for simultaneous MRI reconstruction and segmentation. On a dataset of retrospectively undersampled knee-DESS volumes we demonstrate that irrespective of ultra-high acceleration factors (i.e. 48×) a multitask 3D encoder-decoder is capable of reconstructing with high fidelity the knee MRI, accurately segment cartilaginous and meniscal tissues and reliably provide cartilage thickness. Our multitask solution outperforms two other methods: a compressed sensing reconstruction step, followed by a deep learning-based tissue segmentation. The other method comprises a cascade of two convolutional neural networks that sequentially perform image reconstruction and segmentation.