As with @user3006381 above, my issue was not just relative URLs but that webpack was placing the files as if they were javascript files. Their contents were all basically:
module.exports = __webpack_public_path__ + "7410dd7fd1616d9a61625679285ff5d4.eot";
in the fonts directory instead of the real fonts and the font files were in the output folder under hash codes. To fix this, I had to change the test on my url-loader (in my case my image processor) to not load the fonts folder. I still had to set output.publicPath in webpack.config.js as @will-madden notes in his excellent answer.