The 2019 QSM Reconstruction Challenge compared how different algorithms reconstructed QSM from true local field maps. However, this neglects the importance of residual background fields, and how they degrade the solutions. Here, we used a realistic “total field” simulation to compare three different background field removal methods, and to estimate the susceptibility maps resulting from three state-of-the-art QSM algorithms. We also compared the performance of two single-step algorithms, that use the total field map as an input. Our results showed that the two-step Weak-Harmonic QSM method is robust against residual fields and achieved the lowest error scores, outperforming single-step methods.