We propose Nonlinear Dipole Inversion (NDI) for high-quality Quantitative Susceptibility Mapping (QSM) without additional regularization, while matching the RMSE of state-of-the-art regularized reconstruction techniques. In addition to avoiding over-smoothing these techniques often face, we also obviate the need for parameter selection. NDI is flexible enough to allow for reconstruction from an arbitrary number of head orientations, and outperforms COSMOS using as few as 2-direction data. This is made possible by a nonlinear forward-model that uses the magnitude as an effective prior, for which we derived a simple gradient descent update. We synergistically combine this physics-model with Variational Networks (VN) to leverage the power of deep learning in the VaNDI algorithm. VaNDI adopts this simple gradient descent rule and learns the network parameters during training, hence requires no additional parameter tuning.