Late gadolinium enhancement cardiac magnetic resonance (LGE-CMR) has enabled the accurate myocardial tissue characterization. Due to practical considerations, the acquisition of anisotropic two-dimensional (2D) stack volumes, with low through-plane resolution, still prevails in the clinical routine. We propose a deep learning-based method for reconstructing a super-resolved three-dimensional LGE-CMR data-set from a low resolution 2D short-axis stack volume. The method directly learns the residuals between the high and low resolution images. Results on clinical data-sets show that the proposed technique outperforms the state-of-the-art with regard to image quality. The fast speed of our model furthers facilitates its adoption for practical usage.