We propose a novel deep learning method, Multi-modal Spatial Disentanglement Network (MMSDNet), to segment anatomy in medical images. MMSDNet takes advantage of complementary information provided by multiple sequences of the same patient. Even when trained without annotations, it can segment anatomy (e.g., myocardium) in Late Gadolinium Enhancement (LGE) images, which is essential for assessing myocardial infarction. This is achieved by transferring knowledge from the simultaneously acquired cine-MR data where annotations are easier to be obtained. MMSDNet outperforms classical methods including non-linear registration, and simple copying of contours, as well as the state-of-the-art U-Net model.