The suggested causes are now proven to be mostly impossible. I'm running SSMS V17.9.2 against SS 2014 and still have the problem. Memory problems have existed with this tool since at least 2006 when I started using SSMS.
Yes, MS 'wants' to get rid of diagramming but users won't let them. I have a feeling they will never fix any of these issues because they want users to be so fed up with the tool that enough of them quit using it and they can abandon it entirely.
Restarting is still a workaround if you can stand doing so numerous times per day.