I was able to achieve this by defining a drawable (something like blank_white_shape.xml):
<shape xmlns:android="http://schemas.android.com/apk/res/android"
android:shape="rectangle">
<solid android:color="@android:color/white"/>
</shape>
Then when I want to clear the image view I just call
imageView.setImage(R.drawable.blank_white_shape);
This works beautifully for me!