Plain JavaScript
If a DOM element which is removed is reference-free (no references pointing to it) then yes - the element itself is picked up by the garbage collector as well as any event handlers/listeners associated with it.
var a = document.createElement('div');
var b = document.createElement('p');
// Add event listeners to b etc...
a.appendChild(b);
a.removeChild(b);
b = null;
// A reference to 'b' no longer exists
// Therefore the element and any event listeners attached to it are removed.
However; if there are references that still point to said element, the element and its event listeners are retained in memory.
var a = document.createElement('div');
var b = document.createElement('p');
// Add event listeners to b etc...
a.appendChild(b);
a.removeChild(b);
// A reference to 'b' still exists
// Therefore the element and any associated event listeners are still retained.
jQuery
It would be fair to assume that the relevant methods in jQuery (such as remove()) would function in the exact same way (considering remove() was written using removeChild() for example).
However, this isn't true; the jQuery library actually has an internal method (which is undocumented and in theory could be changed at any time) called cleanData() (here is what this method looks like) which automatically cleans up all the data/events associated with an element upon removal from the DOM (be this via. remove(), empty(), html("") etc).
Older browsers - specifically older versions of IE - are known to have memory leak issues due to event listeners keeping hold of references to the elements they were attached to.
If you want a more in-depth explanation of the causes, patterns and solutions used to fix legacy IE version memory leaks, I fully recommend you read this MSDN article on Understanding and Solving Internet Explorer Leak Patterns.
A few more articles relevant to this:
Manually removing the listeners yourself would probably be a good habit to get into in this case (only if the memory is that vital to your application and you are actually targeting such browsers).