Might have figured it out... Deleting the app from the device worked for me, as others mentioned before (thanks!).
I think the reason is that the app on the device was actually signed with a separate provisioning profile, specifically a distribution profile in my case.