I had the exact same problem while trying to setup a Gitlab pipeline executed by a Docker runner installed on a Raspberry Pi 4
Using nload
to follow bandwidth usage within Docker runner container while pipeline was cloning the repo i saw the network usage dropped down to a few bytes per seconds..
After a some deeper investigations i figured out that the Raspberry temperature was too high and the network card start to dysfunction above 50° Celsius.
Adding a fan to my Raspberry solved the issue.