An open-source toolbox has been implemented to compare state-of-the-art open-source fat-water separation algorithms over synthetic multi-echo data. Data varied in fat-fraction, B0, SNR, number of echoes and echo spacings. Most algorithms proved to be biased for 3 echoes data. For 5 echoes and more, six algorithms were comparable, but two algorithms proved to be inaccurate. Echo spacing scheme impacted quantitative limits of agreements. For proton-density fat-fraction(PDFF) quantification in the extreme ranges, graph-cut approaches provided similar results while IDEAL-CE provided more reliable results. Interestingly, the toolbox also revealed PDFF/T2* quantification to be sensitive to the choice of the fat spectrum.