Adding responseType to the request that is made from angular is indeed the solution, but for me it didn't work until I've set responseType to blob, not to arrayBuffer. The code is self explanatory:
$http({
method : 'GET',
url : 'api/paperAttachments/download/' + id,
responseType: "blob"
}).then(function successCallback(response) {
console.log(response);
var blob = new Blob([response.data]);
FileSaver.saveAs(blob, getFileNameFromHttpResponse(response));
}, function errorCallback(response) {
});