I think Manoj Gupta had the best answer from what I'm seeing. However, the remove command doesn't get rid of any configuration folders or files that may be leftover. Use:
sudo apt-get purge --auto-remove nodejs
The purge command should remove the package and then clean up any configuration files. (see this question for more info on the difference between purge and remove). The auto-remove flag will do the same for packages that were installed by NodeJS.
See the accepted answer on this question for a better explanation.
Although don't forget to handle NPM! Josh's answer covers that.