Based on Farhan Sahibole's answer:
header('Content-Disposition: attachment; filename=Image.png');
header('Content-Type: application/octet-stream'); // Downloading on Android might fail without this
ob_clean();
readfile($file);
This is all I needed for this to work. I stripped off anything that isn't required for this to work.
Key is to use ob_clean();