Dynamically Loading Python Modules
Python is a great language. One of the projects that I am using Python for is on a game server. The Python engine is embedded into a C++ server. One of the main advantages to this is that you can change the Python code without having to re-compile the C++ code.
An even bigger advantage is that Python code can be reloaded without even shutting the Python engine down. Python can reload changed code on the fly.
Python can load code in two different ways.
> Load a string
> Load a module
Loading a String
To load a string, you simply put your code into a string variable and tell Python to run it. In this way you can run a function or class declaration without actually having it execute (unless you place a call to it). This will cause the Python engine to replace the current definitions (if there are any) with the new code.
The Pros: This is very easy. We used this for a while. Our server stores the Python code in a database, so this was the easiest way to extract the code and load it into the Python engine. To reload the changes, you simply re-ran the same load code.
The Cons: This method will over write global variables in your code which began to cause some tricky bugs after a reload. Also, there is no way to UNLOAD the code. It is loaded into the global name space and therefore writing an unload function would be quite difficult. Lastly, because everything is loaded in the same name space, you end up with possible name conflicts.
Load a module
With this method, you tell python to load the module. The only problem is that Python likes to look to the file system for modules. This makes it impossible to load a module from a database. That is what this article is about. We had to override some of the functionality to get it to pull from our database. By doing so, we can now load modules from both the file system (e.g. built in modules) and the database.
The Pros: We can now put our code in one or more modules. The modules have their own name space and can be loaded and unloaded completely with ease. Since modules have their own name space, they don’t trash global variables in the main area. Most importantly, we can have our modules stored in a database. This is important as it allows us to keep all our changes and code in one spot and makes the install and backup easier and allows us to have multiple people work remotely through our custom GUI interface.
The Cons: Reloading on the fly always has the potential to introduce bugs. I am sure we will find more as we go.
The code
Here is the code I use. Note that there is a function called GetModuleCode() that loads the module. This is just a stub here, so replace its code with the code you need to load the module. If you have no problem loading from modules on the hard drive, you might consider just putting them in the Python folder and saving yourself the trouble.
компютри втора употреба
# ========================================================================
# Custom Module Loader
# ========================================================================
# Install the custom module loader, this loader can pull DB modules from the server
import __builtin__
bi = __builtin__.__import__ # Save old import function
import new
import sys
import StringIO
#module loader code references
# http://svn.python.org/projects/python/trunk/Lib/ihooks.py
# http://www.thescripts.com/forum/thread23917.html
# Redefine the __import__ function
# (self, name, globals=None, locals=None, fromlist=[]):
def __import__(name, *args):
# Has this module already been loaded? If so, move on
for mod in sys.modules.keys():
if (mod == name):
# module found
#print "Module already loaded (" + mod + ")"
return sys.modules[mod]
code = GetModuleCode(name)
if code == '':
# Didn't find the module in the database, look for it in the normal python system
return bi(name, *args)
# Found code in the database, load it here
mod = new.module(name)
try:
exec code in mod.__dict__
except:
# Module failed to load? Unload it?
#a=1
#d = self.hooks.modules_dict()
#if name in d:
# del d[name]
del sys.modules[name]
#raise
return mod
#Set __import__ to our __import function
__builtin__.__import__ = __import__
def LoadModule(name):
# Call the import function
__import__(name)
def ReloadModule(name):
UnloadModule(name)
LoadModule(name)
#print "Reload Module (" + name + ")"
def UnloadModule(name):
if (sys.modules.has_key(name)):
#print "Remove Module (" + name + ")"
del sys.modules[name]
def ReloadAllModules():
# Get the list of modules from the database and load them all
sql = "SELECT * FROM your_module_table"
rs = database.Query(sql)
for i in rs:
ReloadModule(i["module_name"])
def GetModuleCode(name):
# Check the database for this module - put your code to retrive your module here
return "python code"
