I have a long-running Python server and would like to be able to upgrade a service without restarting the server. What's the best way do do this?
if foo.py has changed:
unimport foo <-- How do I do this?
import foo
myfoo = foo.Foo()
This question is related to
python
module
reload
python-import
Another way could be to import the module in a function. This way when the function completes the module gets garbage collected.
The following code allows you Python 2/3 compatibility:
try:
reload
except NameError:
# Python 3
from imp import reload
The you can use it as reload() in both versions which makes things simpler.
For Python 2 use built-in function reload():
reload(module)
For Python 2 and 3.2–3.3 use reload from module imp:
import imp
imp.reload(module)
But imp is deprecated since version 3.4 in favor of importlib, so use:
import importlib
importlib.reload(module)
or
from importlib import reload
reload(module)
If you are not in a server, but developing and need to frequently reload a module, here's a nice tip.
First, make sure you are using the excellent IPython shell, from the Jupyter Notebook project. After installing Jupyter, you can start it with ipython, or jupyter console, or even better, jupyter qtconsole, which will give you a nice colorized console with code completion in any OS.
Now in your shell, type:
%load_ext autoreload
%autoreload 2
Now, every time you run your script, your modules will be reloaded.
Beyond the 2, there are other options of the autoreload magic:
%autoreload
Reload all modules (except those excluded by %aimport) automatically now.
%autoreload 0
Disable automatic reloading.
%autoreload 1
Reload all modules imported with %aimport every time before executing the Python code typed.
%autoreload 2
Reload all modules (except those excluded by %aimport) every time before
executing the Python code typed.
Sure, it will also work on a Jupyter Notebook.
For those like me who want to unload all modules (when running in the Python interpreter under Emacs):
for mod in sys.modules.values():
reload(mod)
More information is in Reloading Python modules.
If you encounter the following error, this answer may help you to get a solution:
Traceback (most recent call last): File "FFFF", line 1, in NameError: name 'YYYY' is not defined
OR
Traceback (most recent call last):
File "FFFF", line 1, in
File "/usr/local/lib/python3.7/importlib/__init__.py", line 140, in reload
raise TypeError("reload() argument must be a module")
TypeError: reload() argument must be a module
In case you have an import like the one below, you may need to use the sys.modules to get the module you want to reload:
import importlib
import sys
from YYYY.XXX.ZZZ import CCCC
import AAA.BBB.CC
def reload(full_name)
if full_name in sys.modules:
importlib.reload(sys.modules[full_name])
reload('YYYY.XXX.ZZZ') # this is fine in both cases
reload('AAA.BBB.CC')
importlib.reload(YYYY.XXX.ZZZ) # in my case: this fails
importlib.reload(AAA.BBB.CC) # and this is ok
The main issue is that the importlib.reload accepts module only not string.
It can be especially difficult to delete a module if it is not pure Python.
Here is some information from: How do I really delete an imported module?
You can use sys.getrefcount() to find out the actual number of references.
>>> import sys, empty, os
>>> sys.getrefcount(sys)
9
>>> sys.getrefcount(os)
6
>>> sys.getrefcount(empty)
3
Numbers greater than 3 indicate that it will be hard to get rid of the module. The homegrown "empty" (containing nothing) module should be garbage collected after
>>> del sys.modules["empty"]
>>> del empty
as the third reference is an artifact of the getrefcount() function.
This is the modern way of reloading a module:
from importlib import reload
If you want to support versions of Python older than 3.5, try this:
from sys import version_info
if version_info[0] < 3:
pass # Python 2 has built in reload
elif version_info[0] == 3 and version_info[1] <= 4:
from imp import reload # Python 3.0 - 3.4
else:
from importlib import reload # Python 3.5+
To use it, run reload(MODULE), replacing MODULE with the module you want to reload.
For example, reload(math) will reload the math module.
reload(module), but only if it's completely stand-alone. If anything else has a reference to the module (or any object belonging to the module), then you'll get subtle and curious errors caused by the old code hanging around longer than you expected, and things like isinstance not working across different versions of the same code.
If you have one-way dependencies, you must also reload all modules that depend on the reloaded module to get rid of all the references to the old code. And then reload modules that depend on the reloaded modules, recursively.
If you have circular dependencies, which is very common for example when you are dealing with reloading a package, you must unload all the modules in the group in one go. You can't do this with reload() because it will re-import each module before its dependencies have been refreshed, allowing old references to creep into new modules.
The only way to do it in this case is to hack sys.modules, which is kind of unsupported. You'd have to go through and delete each sys.modules entry you wanted to be reloaded on next import, and also delete entries whose values are None to deal with an implementation issue to do with caching failed relative imports. It's not terribly nice but as long as you have a fully self-contained set of dependencies that doesn't leave references outside its codebase, it's workable.
It's probably best to restart the server. :-)
It can be especially difficult to delete a module if it is not pure Python.
Here is some information from: How do I really delete an imported module?
You can use sys.getrefcount() to find out the actual number of references.
>>> import sys, empty, os
>>> sys.getrefcount(sys)
9
>>> sys.getrefcount(os)
6
>>> sys.getrefcount(empty)
3
Numbers greater than 3 indicate that it will be hard to get rid of the module. The homegrown "empty" (containing nothing) module should be garbage collected after
>>> del sys.modules["empty"]
>>> del empty
as the third reference is an artifact of the getrefcount() function.
Other option. See that Python default importlib.reload will just reimport the library passed as an argument. It won't reload the libraries that your lib import. If you changed a lot of files and have a somewhat complex package to import, you must do a deep reload.
If you have IPython or Jupyter installed, you can use a function to deep reload all libs:
from IPython.lib.deepreload import reload as dreload
dreload(foo)
If you don't have Jupyter, install it with this command in your shell:
pip3 install jupyter
Removing modules from sys.modules requires 'None' types to be deleted as well.
Method 1:
import sys
import json ## your module
for mod in [ m for m in sys.modules if m.lstrip('_').startswith('json') or sys.modules[m] == None ]: del sys.modules[mod]
print( json.dumps( [1] ) ) ## test if functionality has been removed
Method 2, using bookkeeping entries, to remove all dependencies:
import sys
before_import = [mod for mod in sys.modules]
import json ## your module
after_import = [mod for mod in sys.modules if mod not in before_import]
for mod in [m for m in sys.modules if m in after_import or sys.modules[m] == None]: del sys.modules[mod]
print( json.dumps( [2] ) ) ## test if functionality has been removed
Optional, just to be certain all entries are out, if you so choose:
import gc
gc.collect()
In Python 3.0–3.3 you would use: imp.reload(module)
The BDFL has answered this question.
However, imp was deprecated in 3.4, in favour of importlib (thanks @Stefan!).
I think, therefore, you’d now use importlib.reload(module), although I’m not sure.
Another way could be to import the module in a function. This way when the function completes the module gets garbage collected.
Other option. See that Python default importlib.reload will just reimport the library passed as an argument. It won't reload the libraries that your lib import. If you changed a lot of files and have a somewhat complex package to import, you must do a deep reload.
If you have IPython or Jupyter installed, you can use a function to deep reload all libs:
from IPython.lib.deepreload import reload as dreload
dreload(foo)
If you don't have Jupyter, install it with this command in your shell:
pip3 install jupyter
If you are not in a server, but developing and need to frequently reload a module, here's a nice tip.
First, make sure you are using the excellent IPython shell, from the Jupyter Notebook project. After installing Jupyter, you can start it with ipython, or jupyter console, or even better, jupyter qtconsole, which will give you a nice colorized console with code completion in any OS.
Now in your shell, type:
%load_ext autoreload
%autoreload 2
Now, every time you run your script, your modules will be reloaded.
Beyond the 2, there are other options of the autoreload magic:
%autoreload
Reload all modules (except those excluded by %aimport) automatically now.
%autoreload 0
Disable automatic reloading.
%autoreload 1
Reload all modules imported with %aimport every time before executing the Python code typed.
%autoreload 2
Reload all modules (except those excluded by %aimport) every time before
executing the Python code typed.
Sure, it will also work on a Jupyter Notebook.
if 'myModule' in sys.modules:
del sys.modules["myModule"]
Those who are using python 3 and reload from importlib.
If you have problems like it seems that module doesn't reload... That is because it needs some time to recompile pyc (up to 60 sec).I writing this hint just that you know if you have experienced this kind of problem.
reload(module), but only if it's completely stand-alone. If anything else has a reference to the module (or any object belonging to the module), then you'll get subtle and curious errors caused by the old code hanging around longer than you expected, and things like isinstance not working across different versions of the same code.
If you have one-way dependencies, you must also reload all modules that depend on the reloaded module to get rid of all the references to the old code. And then reload modules that depend on the reloaded modules, recursively.
If you have circular dependencies, which is very common for example when you are dealing with reloading a package, you must unload all the modules in the group in one go. You can't do this with reload() because it will re-import each module before its dependencies have been refreshed, allowing old references to creep into new modules.
The only way to do it in this case is to hack sys.modules, which is kind of unsupported. You'd have to go through and delete each sys.modules entry you wanted to be reloaded on next import, and also delete entries whose values are None to deal with an implementation issue to do with caching failed relative imports. It's not terribly nice but as long as you have a fully self-contained set of dependencies that doesn't leave references outside its codebase, it's workable.
It's probably best to restart the server. :-)
This is the modern way of reloading a module:
from importlib import reload
If you want to support versions of Python older than 3.5, try this:
from sys import version_info
if version_info[0] < 3:
pass # Python 2 has built in reload
elif version_info[0] == 3 and version_info[1] <= 4:
from imp import reload # Python 3.0 - 3.4
else:
from importlib import reload # Python 3.5+
To use it, run reload(MODULE), replacing MODULE with the module you want to reload.
For example, reload(math) will reload the math module.
for me for case of Abaqus it is the way it works. Imagine your file is Class_VerticesEdges.py
sys.path.append('D:\...\My Pythons')
if 'Class_VerticesEdges' in sys.modules:
del sys.modules['Class_VerticesEdges']
print 'old module Class_VerticesEdges deleted'
from Class_VerticesEdges import *
reload(sys.modules['Class_VerticesEdges'])
The accepted answer doesn't handle the from X import Y case. This code handles it and the standard import case as well:
def importOrReload(module_name, *names):
import sys
if module_name in sys.modules:
reload(sys.modules[module_name])
else:
__import__(module_name, fromlist=names)
for name in names:
globals()[name] = getattr(sys.modules[module_name], name)
# use instead of: from dfly_parser import parseMessages
importOrReload("dfly_parser", "parseMessages")
In the reloading case, we reassign the top level names to the values stored in the newly reloaded module, which updates them.
For those like me who want to unload all modules (when running in the Python interpreter under Emacs):
for mod in sys.modules.values():
reload(mod)
More information is in Reloading Python modules.
I got a lot of trouble trying to reload something inside Sublime Text, but finally I could wrote this utility to reload modules on Sublime Text based on the code sublime_plugin.py uses to reload modules.
This below accepts you to reload modules from paths with spaces on their names, then later after reloading you can just import as you usually do.
def reload_module(full_module_name):
"""
Assuming the folder `full_module_name` is a folder inside some
folder on the python sys.path, for example, sys.path as `C:/`, and
you are inside the folder `C:/Path With Spaces` on the file
`C:/Path With Spaces/main.py` and want to re-import some files on
the folder `C:/Path With Spaces/tests`
@param full_module_name the relative full path to the module file
you want to reload from a folder on the
python `sys.path`
"""
import imp
import sys
import importlib
if full_module_name in sys.modules:
module_object = sys.modules[full_module_name]
module_object = imp.reload( module_object )
else:
importlib.import_module( full_module_name )
def run_tests():
print( "\n\n" )
reload_module( "Path With Spaces.tests.semantic_linefeed_unit_tests" )
reload_module( "Path With Spaces.tests.semantic_linefeed_manual_tests" )
from .tests import semantic_linefeed_unit_tests
from .tests import semantic_linefeed_manual_tests
semantic_linefeed_unit_tests.run_unit_tests()
semantic_linefeed_manual_tests.run_manual_tests()
if __name__ == "__main__":
run_tests()
If you run for the first time, this should load the module, but if later you can again the method/function run_tests() it will reload the tests files. With Sublime Text (Python 3.3.6) this happens a lot because its interpreter never closes (unless you restart Sublime Text, i.e., the Python3.3 interpreter).
Enthought Traits has a module that works fairly well for this. https://traits.readthedocs.org/en/4.3.0/_modules/traits/util/refresh.html
It will reload any module that has been changed, and update other modules and instanced objects that are using it. It does not work most of the time with __very_private__ methods, and can choke on class inheritance, but it saves me crazy amounts of time from having to restart the host application when writing PyQt guis, or stuff that runs inside programs such as Maya or Nuke. It doesn't work maybe 20-30 % of the time, but it's still incredibly helpful.
Enthought's package doesn't reload files the moment they change - you have to call it explicitely - but that shouldn't be all that hard to implement if you really need it
If you encounter the following error, this answer may help you to get a solution:
Traceback (most recent call last): File "FFFF", line 1, in NameError: name 'YYYY' is not defined
OR
Traceback (most recent call last):
File "FFFF", line 1, in
File "/usr/local/lib/python3.7/importlib/__init__.py", line 140, in reload
raise TypeError("reload() argument must be a module")
TypeError: reload() argument must be a module
In case you have an import like the one below, you may need to use the sys.modules to get the module you want to reload:
import importlib
import sys
from YYYY.XXX.ZZZ import CCCC
import AAA.BBB.CC
def reload(full_name)
if full_name in sys.modules:
importlib.reload(sys.modules[full_name])
reload('YYYY.XXX.ZZZ') # this is fine in both cases
reload('AAA.BBB.CC')
importlib.reload(YYYY.XXX.ZZZ) # in my case: this fails
importlib.reload(AAA.BBB.CC) # and this is ok
The main issue is that the importlib.reload accepts module only not string.
reload(module), but only if it's completely stand-alone. If anything else has a reference to the module (or any object belonging to the module), then you'll get subtle and curious errors caused by the old code hanging around longer than you expected, and things like isinstance not working across different versions of the same code.
If you have one-way dependencies, you must also reload all modules that depend on the reloaded module to get rid of all the references to the old code. And then reload modules that depend on the reloaded modules, recursively.
If you have circular dependencies, which is very common for example when you are dealing with reloading a package, you must unload all the modules in the group in one go. You can't do this with reload() because it will re-import each module before its dependencies have been refreshed, allowing old references to creep into new modules.
The only way to do it in this case is to hack sys.modules, which is kind of unsupported. You'd have to go through and delete each sys.modules entry you wanted to be reloaded on next import, and also delete entries whose values are None to deal with an implementation issue to do with caching failed relative imports. It's not terribly nice but as long as you have a fully self-contained set of dependencies that doesn't leave references outside its codebase, it's workable.
It's probably best to restart the server. :-)
It can be especially difficult to delete a module if it is not pure Python.
Here is some information from: How do I really delete an imported module?
You can use sys.getrefcount() to find out the actual number of references.
>>> import sys, empty, os
>>> sys.getrefcount(sys)
9
>>> sys.getrefcount(os)
6
>>> sys.getrefcount(empty)
3
Numbers greater than 3 indicate that it will be hard to get rid of the module. The homegrown "empty" (containing nothing) module should be garbage collected after
>>> del sys.modules["empty"]
>>> del empty
as the third reference is an artifact of the getrefcount() function.
For Python 2 use built-in function reload():
reload(module)
For Python 2 and 3.2–3.3 use reload from module imp:
import imp
imp.reload(module)
But imp is deprecated since version 3.4 in favor of importlib, so use:
import importlib
importlib.reload(module)
or
from importlib import reload
reload(module)
The following code allows you Python 2/3 compatibility:
try:
reload
except NameError:
# Python 3
from imp import reload
The you can use it as reload() in both versions which makes things simpler.
It can be especially difficult to delete a module if it is not pure Python.
Here is some information from: How do I really delete an imported module?
You can use sys.getrefcount() to find out the actual number of references.
>>> import sys, empty, os
>>> sys.getrefcount(sys)
9
>>> sys.getrefcount(os)
6
>>> sys.getrefcount(empty)
3
Numbers greater than 3 indicate that it will be hard to get rid of the module. The homegrown "empty" (containing nothing) module should be garbage collected after
>>> del sys.modules["empty"]
>>> del empty
as the third reference is an artifact of the getrefcount() function.
The solution from before is good for just getting the reset information, but it will not change all the references (more than reload but less then required). To actually set all the references as well, I had to go into the garbage collector, and rewrite the references there. Now it works like a charm!
Note that this will not work if the GC is turned off, or if reloading data that's not monitored by the GC. If you don't want to mess with the GC, the original answer might be enough for you.
New code:
import importlib
import inspect
import gc
from weakref import ref
def reset_module(module, inner_modules_also=True):
"""
This function is a stronger form of importlib's `reload` function. What it does, is that aside from reloading a
module, it goes to the old instance of the module, and sets all the (not read-only) attributes, functions and classes
to be the reloaded-module's
:param module: The module to reload (module reference, not the name)
:param inner_modules_also: Whether to treat ths module as a package as well, and reload all the modules within it.
"""
# For the case when the module is actually a package
if inner_modules_also:
submods = {submod for _, submod in inspect.getmembers(module)
if (type(submod).__name__ == 'module') and (submod.__package__.startswith(module.__name__))}
for submod in submods:
reset_module(submod, True)
# First, log all the references before reloading (because some references may be changed by the reload operation).
module_tree = _get_tree_references_to_reset_recursively(module, module.__name__)
new_module = importlib.reload(module)
_reset_item_recursively(module, module_tree, new_module)
def _update_referrers(item, new_item):
refs = gc.get_referrers(item)
weak_ref_item = ref(item)
for coll in refs:
if type(coll) == dict:
enumerator = coll.keys()
elif type(coll) == list:
enumerator = range(len(coll))
else:
continue
for key in enumerator:
if weak_ref_item() is None:
# No refs are left in the GC
return
if coll[key] is weak_ref_item():
coll[key] = new_item
def _get_tree_references_to_reset_recursively(item, module_name, grayed_out_item_ids = None):
if grayed_out_item_ids is None:
grayed_out_item_ids = set()
item_tree = dict()
attr_names = set(dir(item)) - _readonly_attrs
for sub_item_name in attr_names:
sub_item = getattr(item, sub_item_name)
item_tree[sub_item_name] = [sub_item, None]
try:
# Will work for classes and functions defined in that module.
mod_name = sub_item.__module__
except AttributeError:
mod_name = None
# If this item was defined within this module, deep-reset
if (mod_name is None) or (mod_name != module_name) or (id(sub_item) in grayed_out_item_ids) \
or isinstance(sub_item, EnumMeta):
continue
grayed_out_item_ids.add(id(sub_item))
item_tree[sub_item_name][1] = \
_get_tree_references_to_reset_recursively(sub_item, module_name, grayed_out_item_ids)
return item_tree
def _reset_item_recursively(item, item_subtree, new_item):
# Set children first so we don't lose the current references.
if item_subtree is not None:
for sub_item_name, (sub_item, sub_item_tree) in item_subtree.items():
try:
new_sub_item = getattr(new_item, sub_item_name)
except AttributeError:
# The item doesn't exist in the reloaded module. Ignore.
continue
try:
# Set the item
_reset_item_recursively(sub_item, sub_item_tree, new_sub_item)
except Exception as ex:
pass
_update_referrers(item, new_item)
As written in @bobince's answer, if there's already a reference to that module in another module (especially if it was imported with the as keyword like import numpy as np), that instance will not be overwritten.
This proved quite problematic to me when applying tests that required a "clean-slate" state of the configuration modules, so I've written a function named reset_module that uses importlib's reload function and recursively overwrites all the declared module's attributes. It has been tested with Python version 3.6.
import importlib
import inspect
from enum import EnumMeta
_readonly_attrs = {'__annotations__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__',
'__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__func__', '__ge__', '__get__',
'__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__',
'__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__',
'__reduce__', '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__sizeof__', '__str__',
'__subclasshook__', '__weakref__', '__members__', '__mro__', '__itemsize__', '__isabstractmethod__',
'__basicsize__', '__base__'}
def reset_module(module, inner_modules_also=True):
"""
This function is a stronger form of importlib's `reload` function. What it does, is that aside from reloading a
module, it goes to the old instance of the module, and sets all the (not read-only) attributes, functions and classes
to be the reloaded-module's
:param module: The module to reload (module reference, not the name)
:param inner_modules_also: Whether to treat ths module as a package as well, and reload all the modules within it.
"""
new_module = importlib.reload(module)
reset_items = set()
# For the case when the module is actually a package
if inner_modules_also:
submods = {submod for _, submod in inspect.getmembers(module)
if (type(submod).__name__ == 'module') and (submod.__package__.startswith(module.__name__))}
for submod in submods:
reset_module(submod, True)
_reset_item_recursively(module, new_module, module.__name__, reset_items)
def _reset_item_recursively(item, new_item, module_name, reset_items=None):
if reset_items is None:
reset_items = set()
attr_names = set(dir(item)) - _readonly_attrs
for sitem_name in attr_names:
sitem = getattr(item, sitem_name)
new_sitem = getattr(new_item, sitem_name)
try:
# Set the item
setattr(item, sitem_name, new_sitem)
try:
# Will work for classes and functions defined in that module.
mod_name = sitem.__module__
except AttributeError:
mod_name = None
# If this item was defined within this module, deep-reset
if (mod_name is None) or (mod_name != module_name) or (id(sitem) in reset_items) \
or isinstance(sitem, EnumMeta): # Deal with enums
continue
reset_items.add(id(sitem))
_reset_item_recursively(sitem, new_sitem, module_name, reset_items)
except Exception as ex:
raise Exception(sitem_name) from ex
Note: Use with care! Using these on non-peripheral modules (modules that define externally-used classes, for example) might lead to internal problems in Python (such as pickling/un-pickling issues).
for me for case of Abaqus it is the way it works. Imagine your file is Class_VerticesEdges.py
sys.path.append('D:\...\My Pythons')
if 'Class_VerticesEdges' in sys.modules:
del sys.modules['Class_VerticesEdges']
print 'old module Class_VerticesEdges deleted'
from Class_VerticesEdges import *
reload(sys.modules['Class_VerticesEdges'])
reload(module), but only if it's completely stand-alone. If anything else has a reference to the module (or any object belonging to the module), then you'll get subtle and curious errors caused by the old code hanging around longer than you expected, and things like isinstance not working across different versions of the same code.
If you have one-way dependencies, you must also reload all modules that depend on the reloaded module to get rid of all the references to the old code. And then reload modules that depend on the reloaded modules, recursively.
If you have circular dependencies, which is very common for example when you are dealing with reloading a package, you must unload all the modules in the group in one go. You can't do this with reload() because it will re-import each module before its dependencies have been refreshed, allowing old references to creep into new modules.
The only way to do it in this case is to hack sys.modules, which is kind of unsupported. You'd have to go through and delete each sys.modules entry you wanted to be reloaded on next import, and also delete entries whose values are None to deal with an implementation issue to do with caching failed relative imports. It's not terribly nice but as long as you have a fully self-contained set of dependencies that doesn't leave references outside its codebase, it's workable.
It's probably best to restart the server. :-)
I got a lot of trouble trying to reload something inside Sublime Text, but finally I could wrote this utility to reload modules on Sublime Text based on the code sublime_plugin.py uses to reload modules.
This below accepts you to reload modules from paths with spaces on their names, then later after reloading you can just import as you usually do.
def reload_module(full_module_name):
"""
Assuming the folder `full_module_name` is a folder inside some
folder on the python sys.path, for example, sys.path as `C:/`, and
you are inside the folder `C:/Path With Spaces` on the file
`C:/Path With Spaces/main.py` and want to re-import some files on
the folder `C:/Path With Spaces/tests`
@param full_module_name the relative full path to the module file
you want to reload from a folder on the
python `sys.path`
"""
import imp
import sys
import importlib
if full_module_name in sys.modules:
module_object = sys.modules[full_module_name]
module_object = imp.reload( module_object )
else:
importlib.import_module( full_module_name )
def run_tests():
print( "\n\n" )
reload_module( "Path With Spaces.tests.semantic_linefeed_unit_tests" )
reload_module( "Path With Spaces.tests.semantic_linefeed_manual_tests" )
from .tests import semantic_linefeed_unit_tests
from .tests import semantic_linefeed_manual_tests
semantic_linefeed_unit_tests.run_unit_tests()
semantic_linefeed_manual_tests.run_manual_tests()
if __name__ == "__main__":
run_tests()
If you run for the first time, this should load the module, but if later you can again the method/function run_tests() it will reload the tests files. With Sublime Text (Python 3.3.6) this happens a lot because its interpreter never closes (unless you restart Sublime Text, i.e., the Python3.3 interpreter).
2018-02-01
foo must be imported successfully in advance. from importlib import reload, reload(foo) 31.5. importlib — The implementation of import — Python 3.6.4 documentation
Those who are using python 3 and reload from importlib.
If you have problems like it seems that module doesn't reload... That is because it needs some time to recompile pyc (up to 60 sec).I writing this hint just that you know if you have experienced this kind of problem.
if 'myModule' in sys.modules:
del sys.modules["myModule"]
The accepted answer doesn't handle the from X import Y case. This code handles it and the standard import case as well:
def importOrReload(module_name, *names):
import sys
if module_name in sys.modules:
reload(sys.modules[module_name])
else:
__import__(module_name, fromlist=names)
for name in names:
globals()[name] = getattr(sys.modules[module_name], name)
# use instead of: from dfly_parser import parseMessages
importOrReload("dfly_parser", "parseMessages")
In the reloading case, we reassign the top level names to the values stored in the newly reloaded module, which updates them.
The solution from before is good for just getting the reset information, but it will not change all the references (more than reload but less then required). To actually set all the references as well, I had to go into the garbage collector, and rewrite the references there. Now it works like a charm!
Note that this will not work if the GC is turned off, or if reloading data that's not monitored by the GC. If you don't want to mess with the GC, the original answer might be enough for you.
New code:
import importlib
import inspect
import gc
from weakref import ref
def reset_module(module, inner_modules_also=True):
"""
This function is a stronger form of importlib's `reload` function. What it does, is that aside from reloading a
module, it goes to the old instance of the module, and sets all the (not read-only) attributes, functions and classes
to be the reloaded-module's
:param module: The module to reload (module reference, not the name)
:param inner_modules_also: Whether to treat ths module as a package as well, and reload all the modules within it.
"""
# For the case when the module is actually a package
if inner_modules_also:
submods = {submod for _, submod in inspect.getmembers(module)
if (type(submod).__name__ == 'module') and (submod.__package__.startswith(module.__name__))}
for submod in submods:
reset_module(submod, True)
# First, log all the references before reloading (because some references may be changed by the reload operation).
module_tree = _get_tree_references_to_reset_recursively(module, module.__name__)
new_module = importlib.reload(module)
_reset_item_recursively(module, module_tree, new_module)
def _update_referrers(item, new_item):
refs = gc.get_referrers(item)
weak_ref_item = ref(item)
for coll in refs:
if type(coll) == dict:
enumerator = coll.keys()
elif type(coll) == list:
enumerator = range(len(coll))
else:
continue
for key in enumerator:
if weak_ref_item() is None:
# No refs are left in the GC
return
if coll[key] is weak_ref_item():
coll[key] = new_item
def _get_tree_references_to_reset_recursively(item, module_name, grayed_out_item_ids = None):
if grayed_out_item_ids is None:
grayed_out_item_ids = set()
item_tree = dict()
attr_names = set(dir(item)) - _readonly_attrs
for sub_item_name in attr_names:
sub_item = getattr(item, sub_item_name)
item_tree[sub_item_name] = [sub_item, None]
try:
# Will work for classes and functions defined in that module.
mod_name = sub_item.__module__
except AttributeError:
mod_name = None
# If this item was defined within this module, deep-reset
if (mod_name is None) or (mod_name != module_name) or (id(sub_item) in grayed_out_item_ids) \
or isinstance(sub_item, EnumMeta):
continue
grayed_out_item_ids.add(id(sub_item))
item_tree[sub_item_name][1] = \
_get_tree_references_to_reset_recursively(sub_item, module_name, grayed_out_item_ids)
return item_tree
def _reset_item_recursively(item, item_subtree, new_item):
# Set children first so we don't lose the current references.
if item_subtree is not None:
for sub_item_name, (sub_item, sub_item_tree) in item_subtree.items():
try:
new_sub_item = getattr(new_item, sub_item_name)
except AttributeError:
# The item doesn't exist in the reloaded module. Ignore.
continue
try:
# Set the item
_reset_item_recursively(sub_item, sub_item_tree, new_sub_item)
except Exception as ex:
pass
_update_referrers(item, new_item)
As written in @bobince's answer, if there's already a reference to that module in another module (especially if it was imported with the as keyword like import numpy as np), that instance will not be overwritten.
This proved quite problematic to me when applying tests that required a "clean-slate" state of the configuration modules, so I've written a function named reset_module that uses importlib's reload function and recursively overwrites all the declared module's attributes. It has been tested with Python version 3.6.
import importlib
import inspect
from enum import EnumMeta
_readonly_attrs = {'__annotations__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__',
'__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__func__', '__ge__', '__get__',
'__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__init_subclass__',
'__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__qualname__',
'__reduce__', '__reduce_ex__', '__repr__', '__self__', '__setattr__', '__sizeof__', '__str__',
'__subclasshook__', '__weakref__', '__members__', '__mro__', '__itemsize__', '__isabstractmethod__',
'__basicsize__', '__base__'}
def reset_module(module, inner_modules_also=True):
"""
This function is a stronger form of importlib's `reload` function. What it does, is that aside from reloading a
module, it goes to the old instance of the module, and sets all the (not read-only) attributes, functions and classes
to be the reloaded-module's
:param module: The module to reload (module reference, not the name)
:param inner_modules_also: Whether to treat ths module as a package as well, and reload all the modules within it.
"""
new_module = importlib.reload(module)
reset_items = set()
# For the case when the module is actually a package
if inner_modules_also:
submods = {submod for _, submod in inspect.getmembers(module)
if (type(submod).__name__ == 'module') and (submod.__package__.startswith(module.__name__))}
for submod in submods:
reset_module(submod, True)
_reset_item_recursively(module, new_module, module.__name__, reset_items)
def _reset_item_recursively(item, new_item, module_name, reset_items=None):
if reset_items is None:
reset_items = set()
attr_names = set(dir(item)) - _readonly_attrs
for sitem_name in attr_names:
sitem = getattr(item, sitem_name)
new_sitem = getattr(new_item, sitem_name)
try:
# Set the item
setattr(item, sitem_name, new_sitem)
try:
# Will work for classes and functions defined in that module.
mod_name = sitem.__module__
except AttributeError:
mod_name = None
# If this item was defined within this module, deep-reset
if (mod_name is None) or (mod_name != module_name) or (id(sitem) in reset_items) \
or isinstance(sitem, EnumMeta): # Deal with enums
continue
reset_items.add(id(sitem))
_reset_item_recursively(sitem, new_sitem, module_name, reset_items)
except Exception as ex:
raise Exception(sitem_name) from ex
Note: Use with care! Using these on non-peripheral modules (modules that define externally-used classes, for example) might lead to internal problems in Python (such as pickling/un-pickling issues).
2018-02-01
foo must be imported successfully in advance. from importlib import reload, reload(foo) 31.5. importlib — The implementation of import — Python 3.6.4 documentation
In Python 3.0–3.3 you would use: imp.reload(module)
The BDFL has answered this question.
However, imp was deprecated in 3.4, in favour of importlib (thanks @Stefan!).
I think, therefore, you’d now use importlib.reload(module), although I’m not sure.
Removing modules from sys.modules requires 'None' types to be deleted as well.
Method 1:
import sys
import json ## your module
for mod in [ m for m in sys.modules if m.lstrip('_').startswith('json') or sys.modules[m] == None ]: del sys.modules[mod]
print( json.dumps( [1] ) ) ## test if functionality has been removed
Method 2, using bookkeeping entries, to remove all dependencies:
import sys
before_import = [mod for mod in sys.modules]
import json ## your module
after_import = [mod for mod in sys.modules if mod not in before_import]
for mod in [m for m in sys.modules if m in after_import or sys.modules[m] == None]: del sys.modules[mod]
print( json.dumps( [2] ) ) ## test if functionality has been removed
Optional, just to be certain all entries are out, if you so choose:
import gc
gc.collect()
Enthought Traits has a module that works fairly well for this. https://traits.readthedocs.org/en/4.3.0/_modules/traits/util/refresh.html
It will reload any module that has been changed, and update other modules and instanced objects that are using it. It does not work most of the time with __very_private__ methods, and can choke on class inheritance, but it saves me crazy amounts of time from having to restart the host application when writing PyQt guis, or stuff that runs inside programs such as Maya or Nuke. It doesn't work maybe 20-30 % of the time, but it's still incredibly helpful.
Enthought's package doesn't reload files the moment they change - you have to call it explicitely - but that shouldn't be all that hard to implement if you really need it
Source: Stackoverflow.com