We Keep Coding

How to load a dylib-file as CPython-extension?

This has been asked before (e.g. here), but the given solution (i.e. renaming file to *.so) is not acceptable. I have a CPython extension called name.dylib, which cannot be imported. If the filename is changed to use name.so it is imported correctly. Changing the filename is not an option**, and should not be necessary.
Python has a lot of hooks for searching for modules, so there must be a way to make it recognise a dylib-file. Can someone show how to do this? Using a low level import which spells out the whole filename is not nice but is an acceptable solution.
** because the build code forces dylib, and, other contexts I have assume it. The extension module is dual purpose, it can by used both as an ordinary shared library and a Python extension. Using a symlink does work, but is a last resort because it requires manual intervention in an automated process.

You could manipulate sys.path_hooks and replace FileFinder-hook with one which would accept .dylib-extensions. But see also the simpler but less convinient alternative which would import given the full file-name of the extension.
More information how .so, .py and .pyc files are imported can be found for example in this answer of mine.
This manipulation could look like following:
import sys
import importlib
from importlib.machinery import FileFinder, ExtensionFileLoader
# pick right loader for .dylib-files:
dylib_extension = ExtensionFileLoader, ['.dylib']
# add dylib-support to file-extension supported per default
all_supported_loaders = [dylib_extension]+ importlib._bootstrap_external._get_supported_file_loaders()
# replace the last hook (i.e. FileFinder) with one recognizing `.dylib` as well:
sys.path_hooks.pop()
sys.path_hooks.append(FileFinder.path_hook(*all_supported_loaders))
#and now import name.dylib via
import name
This must be the first code executed, when python-script starts to run. Other modules might not expect sys.path_hooks being manipulated somewhere during the run of the program, so there might be some problems with other modules (like pdb, traceback and so). For example:
import pdb
#above code
import name
will fail, while
#above code
import pdb
import name
will work, as pdb seems to manipulate the import-machinery.
Normally, FileFinder-hook is the last in the sys.path_hooks, because it is the last resort, once path_hook_for_FileFinder is called for a path, the finder is returned (ImportError should be raised if PathFinder should look at further hooks):
def path_hook_for_FileFinder(path):
"""Path hook for importlib.machinery.FileFinder."""
if not _path_isdir(path):
raise ImportError('only directories are supported', path=path)
return cls(path, *loader_details) # HERE Finder is returned!
However, one might want to be sure and check, that really the right hook is replaced.
A simpler alternative would be to use imp.load_dynamic (neglecting for the moment that imp is depricated):
import imp
imp.load_dynamic('name', 'name.dylib') # or what ever path is used
That might be more robust than the first solution (no problems with pdb for example) but less convinient for bigger projects.

Make modules available globally

I'm trying to make my modules available globally
Filesystem structure
main.py
module_static.py
folder/module_dynamic.py # this is example, but imagine 100s of modules
main.py
print('Loading module_static')
import module_static
module_static.test()
# Trying to make module_static available globally
globals()['module_static'] = module_static
__all__ = ['module_static']
print('Loading module_dynamic')
import sys
sys.path.append('./folder/')
import module_dynamic
module_dynamic.test()
module_static.py
def test():
print(' -> This is module_static')
module_dynamic.py
def test():
print(' -> This is module_dynamic')
module_static.test()
Running main.py creates the following execution flow main.py -> module_dynamic.py -> module_static.py
So as you can see:
Loading of modules is working properly
However, despite trying to make module_static available globally, it isn't working a module_dynamic.py throws an error saying module_static doesn't exist
How can I make module_static.py available in module_dynamic.py (ideally without having to write any additional code in module_dynamic.py)?

Not saying it's good practice, but you can do
main.py
import builtins
import module_static
builtins.module_static = module_static
This should allow you to use module_static from anywhere.
More info on builtins: How to make a cross-module variable?

It can't work the way you expect. globals() return a dict of globals variables in your script. Maybe this may help you to understand
You can take a look at this course for better understanding
https://www.python-course.eu/python3_global_vs_local_variables.php
Anyway, you will have to import the module to use it.
If it's just a local tool for your personnal use, you could move it to the folder
{Python_installation_folder}/Lib.
Then, in any script, you will be able to do
import module_static
and use your module.
If you want to share your module with other people, publish (upload) it on PyPi. You could follow the tutorial bellow
https://anweshadas.in/how-to-upload-a-package-in-pypi-using-twine/

Getting paths to code of imported modules

I have to analyse some python file. For this I should analyse all modules (i.e. get source code of this modules if they written on Python) imported in given file.
How I can get paths to files with imported python-modules?
I try to use sys.path, but it gives all paths, where does python interpreter may search modules

For the 3rd party modules, following should print the file path.
module_file = <module_name>.__file__
You can then print the file contents.

Good question. I tried some things and the problem is that many standard modules seem quite inaccessible, like the math module isn't even Python, but a C library (.so file). If you only need to access user-defined modules and not standard ones, something like this could get you files and source:
import inspect
import sys
def main():
# sys.modules contains a mapping between module names and module
# objects, but many more than the one imported in a file. dir()
# returns a list of names available to the local scope (also variables
# functions etc.). Combine those two and you get the modules available
# to the local scope
modules = [sys.modules[i] for i in dir() if i in sys.modules]
files = []
code = []
for module in modules:
try:
# modules may have a __file__ attribute
files.append(module.__file__)
# get's you the actual code
code.append(inspect.getsource(module))
except:
pass
print(files, code)
if __name__ == '__main__':
main()
__file__ is not defined for all standard modules and inspect.getsource() doesn't work for them either, that's why the try-except block, but for non-standard modules this could get you started.

Modules are found in /usr/lib/python3/dist-packages in unix-like systems
and are found in C:\Python34\Lib in windows

How do I debug a "can not import" error on package import

I am new to python and trying to get a feel for python fuse with this tutorial. I installed pythonfuse with pip. I installed os x fuse by downloading a dmg and installing on os x. When I run this line of code from fuse import FUSE, FuseOSError, Operations from the tutorial I see this:
akh2103$ python myfuse.py
Traceback (most recent call last):
File "myfuse.py", line 10, in <module>
from fuse import FUSE, FuseOSError, Operations
ImportError: cannot import name FUSE
It seems like it can't find the fuse package, can't find the python fuse package or can't find the FUSE, FuseOSError and Operations methods within the package. Which one is it? When I type import fuse where does Python go to look for the fuse package? I'm used to class paths in java: is there a python equivalent? I'm very new to python. How do I begin to debug this.

It looks in /Library/Python/<version>/site-packages.
You may be having multiple versions which may be the cause of the problem.
Find out where pip installed fuse.
You can use the PYTHONPATH environment variable to add additional folders.

The fuse module was found (otherwise you would see "No module named fuse"). The error you got means that "FUSE" symbol is not found in the module fuse.
My guess is there are several python bindings for FUSE and you are probably looking at a tutorial for a different module than the one you are loading. The other alternative is some drastic changes in the library between different versions.
If you want to see all the symbols exported by a module, use dir():
import fuse
dir(fuse)

Say this was your directory structure:
myapp/
firstpackage/
__init__.py
firstmodule.py
secondpackage/
__init__.py
secondmodule.py
__init__.py
myfirstapp.py
firstmodule.py
def first_function(data):
return data
def second_function(data):
return data
Now let's say we're working from :mod:`myfirstapp`.
If we wanted to access :func:`first_function`, we'd import like:
from myapp.firstpackage.firstmodule import first_function
print first_function('foo')
:mod:`__init__` in 'firstpackage' directory allows :mod:`firstmodule` to be accessed from outside of it's directory. The inclusion of :file:`__init__.py` within a directory makes that directory a Python package.
However, it's better practice to import the whole module like:
import myapp.firstpackage.firstmodule as firstmodule
print firstmodule.first_function('foo')
print firstmodule.second_function('bar')
Another method would be:
from myapp.firstpackage import firstmodule
print firstmodule.second_function('foo')
That way everything is accessible from within your module, and better for readability.
That being said, the :exc:`ImportError` you're receiving is because 'FUSE' does not exist in :mod:`fuse`, whether it's data, class or a function.
Open fuse.py and do a search for 'FUSE' and see if anything comes up. Look for:
def FUSE(): ...
class FUSE(..): ...
FUSE = ...
I know the whole package/module lesson was off topic from your question, but you said you were new, so I thought I'd elaborate :P

Python : from module import * in __init__

Here is a simple case: I want to define a module in python name robot. So, I have a folder named robot with these two files:
__init__.py:
from test import a
test.py:
a = "hello world"
Now, when I import robot in the interpreter, the robot namespace includes test and a. However, I only want it to include a. Why this odd behavior?
EDIT:
Here's a slightly more representative example of what I want to achieve:
Given the following files:
__init__.py:
from spam import a
import ham
spam.py:
a = "hello world"
ham.py:
b = "foo"
Can I have a robot namespace containing a and ham at its top level but not spam?

You have created not just a module but a package. A package contains its submodules in its namespace (once they have been imported, as you imported test here). This is as it should be, since the usual way of using packages is to provide a grouping of several modules. There's not much use to making a package with only one module (i.e., one contentful .py file) inside it.
If you just want a one-file module, just create a file called robots.py and put your code in there.
Edit: See this previous question. The answer is that you should in general not worry about excluding module names from your package namespace. The modules are supposed to be in the package namespace. If you want to add functions and stuff from submodules as well, for convenience, that's fine, but there's not really anything to be gained by "covering your tracks" and hiding the modules you imported. However, as described in the answers to that question, there are some hackish ways to approximate what you want.

Are you just asking how to import specific modules or functions?
test.py:
import robot.spam.a
import robot.ham
Don't import the entire package.