I have several compiled python modules; they are put into a single .so (to avoid runtime linking, there are cross-module symbol dependencies), but a number of symlinks points to this .so:
foo.so -> liball.so
bar.so -> liball.so
liball.so
This way, I can do import foo (Python will call initfoo() defined in liball.so) or import bar (calls initbar()).
I am wondering if this approach will work on Windows?
Probably not, but you could achieve your goal with
import sys
import liball
sys.modules['foo'] = liball
sys.modules['bar'] = liball
if you need to import them at several places, or with
import liball as foo, libalb as bar, liball
if you need that only at one place.
It might be, however, that the distinction between initfoo() and initbar() cannot be held and that both must be done so that the module effectively contains everything to be contained in both modules.
If foo partially contains the same symbols as bar, but with a different meaning, this approach won't work. But then you can just copy the file. This will occupy more disk space than needed, but that doesn't hurt so much, IMHO.
Related
This has been asked before (e.g. here), but the given solution (i.e. renaming file to *.so) is not acceptable. I have a CPython extension called name.dylib, which cannot be imported. If the filename is changed to use name.so it is imported correctly. Changing the filename is not an option**, and should not be necessary.
Python has a lot of hooks for searching for modules, so there must be a way to make it recognise a dylib-file. Can someone show how to do this? Using a low level import which spells out the whole filename is not nice but is an acceptable solution.
** because the build code forces dylib, and, other contexts I have assume it. The extension module is dual purpose, it can by used both as an ordinary shared library and a Python extension. Using a symlink does work, but is a last resort because it requires manual intervention in an automated process.
You could manipulate sys.path_hooks and replace FileFinder-hook with one which would accept .dylib-extensions. But see also the simpler but less convinient alternative which would import given the full file-name of the extension.
More information how .so, .py and .pyc files are imported can be found for example in this answer of mine.
This manipulation could look like following:
import sys
import importlib
from importlib.machinery import FileFinder, ExtensionFileLoader
# pick right loader for .dylib-files:
dylib_extension = ExtensionFileLoader, ['.dylib']
# add dylib-support to file-extension supported per default
all_supported_loaders = [dylib_extension]+ importlib._bootstrap_external._get_supported_file_loaders()
# replace the last hook (i.e. FileFinder) with one recognizing `.dylib` as well:
sys.path_hooks.pop()
sys.path_hooks.append(FileFinder.path_hook(*all_supported_loaders))
#and now import name.dylib via
import name
This must be the first code executed, when python-script starts to run. Other modules might not expect sys.path_hooks being manipulated somewhere during the run of the program, so there might be some problems with other modules (like pdb, traceback and so). For example:
import pdb
#above code
import name
will fail, while
#above code
import pdb
import name
will work, as pdb seems to manipulate the import-machinery.
Normally, FileFinder-hook is the last in the sys.path_hooks, because it is the last resort, once path_hook_for_FileFinder is called for a path, the finder is returned (ImportError should be raised if PathFinder should look at further hooks):
def path_hook_for_FileFinder(path):
"""Path hook for importlib.machinery.FileFinder."""
if not _path_isdir(path):
raise ImportError('only directories are supported', path=path)
return cls(path, *loader_details) # HERE Finder is returned!
However, one might want to be sure and check, that really the right hook is replaced.
A simpler alternative would be to use imp.load_dynamic (neglecting for the moment that imp is depricated):
import imp
imp.load_dynamic('name', 'name.dylib') # or what ever path is used
That might be more robust than the first solution (no problems with pdb for example) but less convinient for bigger projects.
I'm working on a python project that has some of duplicated code.
I'm trying to remove the duplicated files, and keep only one file, however, my problem is that others projects already use the removes files, and I don't want to have to change their imports.
Example:
src/
-----a.py
-----b.py
Lets assume both a and b implement the same function f, and there are pieces of code in the other project who uses from src.a import f and others use from src.b import f
Right now my solution is to keep the implementation in a and have b only contain from src.a import f.
Is there a different way where I could completely remove b but still have the ability to use from src.b import f ?
Without a horrible import hook hack that emulates a virtual b.py, no.
However, you can just make b.py a "re-export" module like
from a import foo, baz, quux
assuming those are the duplicate names.
What is the best approach to have the same classes coexist in ipython environment.
I want to gradually migrate some classes in hierarchy (starting from the root classes) from Python to Cython.
I want to be able to have both versions running (in ipython env), so I can compare performance, and be able to fall-back when needed.
Is there some other approach that may work even if not exactly the way I want.
For my current experiments I started with renaming the classes and importing them separately.
F.e. :
import pyximport; pyximport.install()
from blah import *
from c_blah import *
b = Blah()
cb = c_Blah()
%timeit -n 1000 b.op()
%timeit -n 1000 cb.op()
that is cumbersome because I had to rename all class-attr/method accesses.
Also this does not solve my dilemma once I go down the hierarchy.
Any other ideas how to approach this ?
I mean incremental recoding in Cython.
cswiercz makes good point :
from blah import Blah
from cblah import Blah as cBlah
this is OK, but supporting hierarchy will require modifications.
Will keep it open for other ideas.
You can use the import x as y method for renaming definitions within modules. For example, if foo.py contains a function func and bar.pyx also contains the function func (perhaps this is you trying to write a Cython version of foo.func()) then you can do the following in your timing script:
# separate imports
from foo import func as func_py
from bar import func as func_cy
# use the code
func_py(2.0)
func_cy(2.0)
This way you can keep a meaningful naming scheme within the foo and bar modules.
I want to import a python module without adding its containing folder to the python path. I would want the import look like
from A import B as C
Due to the specific path that shall be used, the import looks like
import imp
A = imp.load_source('A', 'path')
C = A.B
This is quite unhandy with long paths and module names. Is there an easier way? Is there A way, where the module is not added to the local variables (no A)?
If you just don't want A to be visible at a global level, you could stick the import (imp.load_source) inside a function. If you actually don't want a module object at all in the local scope, you can do that too, but I wouldn't recommend it.
If module A is a python source file you could read in the file (or even just the relevant portion that you want) and run an exec on it.
source.py
MY_GLOBAL_VAR = 1
def my_func():
print 'hello'
Let's say you have some code that wants my_func
path = '/path/to/source.py'
execfile(path)
my_func()
# 'hello'
Be aware that you're also going to get anything else defined in the file (like MY_GLOBAL_VAR). Again, this will work, but I wouldn't recommend it
Someone looking at your code won't be able to see where my_func came from.
You're essentially doing the same thing as a from A import * import, which is generally frowned upon in python, because , you could be importing all sorts of things into your namespace that you didn't want. And even if it works now, if the source code changes, it could import names that shadow your own global symbols.
It's potentially a security hole, since you could be exec'ing an untrusted source file.
It's way more verbose than a regular python import.
Suppose you have the following
b
b/__init__.py
b/c
b/c/__init__.py
b/c/d
b/c/d/__init__.py
In some python packages, if you import b, you only get the symbols defined in b. To access b.c, you have to explicitly import b.c or from b import c. In other words, you have to
import b
import b.c
import b.c.d
print b.c.d
In other cases I saw an automatic import of all the subpackages. This means that the following code does not produce an error
import b
print b.c.d
because b/__init__.py takes care of importing its subpackages.
I tend to prefer the first (explicit better than implicit), and I always used it, but are there cases where the second one is preferred to the first?
I like namespaces -- so I think that import b should only get what's in b itself (presumably in b/__init__.py). If there's a reason to segregate other functionality in b.c, b.c.d, or whatever, then just import b should not drag it all in -- if the "drag it all in" does happen, I think that suggests that the namespace separation was probably a bogus one to start with. Of course, there are examples even in the standard library (import os, then you can use os.path.join and the like), but they're ancient, by now essentially "grandfathered" things from way before the Python packaging system was mature and stable. In new code, I'd strongly recommend that a package should not drag its subpackages along for the ride when you import it. (Do import this at the Python prompt and contemplate the very last line it shows;-).
__all__ = [your vars, functions, classes]
Use syntax above in package b's __init__.py to auto load things listed in dict. :)