No issues with the latest kartoza/qgis-desktop
I ran
docker pull kartoza/qgis-desktop
followed by
docker run -it --rm --name "qgis-desktop-2-4" -v ${HOME}:/home/${USER} -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY kartoza/qgis-desktop:latest
I did try multiple times without the conflict error - you do have to exit the app beforehand. Also, please note the parameters do differ slightly.