Since at the time of installing of any module it mainly put corresponding .pm files in respective directories.
So if you want to remove module only for some testing purpose or temporarily best is to find the path where module is stored using perldoc -l <MODULE>
and then simply move the module from there to some other location.
This approach can also be tried as a more permanent solution but i am not aware of any negative consequences as i do it mainly for testing.