I have a compiled a Python extension.
The resulting binary mylib.so file can be imported in the Python script and works fine.
Now I am wondering how to write the interface stub file mylib.pyi such, that pylint and the Python language server used in VS Code can use it?
For now the native library is just in the root of my scripts.
When putting the mylib.pyi next to it pylint ignores it.
I know this question is a year old but the answer might help the next person. I managed to do this by writing the following setup file:
from distutils.core import setup, Extension
extension = Extension(
name = 'dummy',
sources = ...,
...
)
setup(
name = 'dummy',
ext_modules = [extension],
packages = ['dummy'],
package_dir = {'dummy' : './stub'},
package_data = {
'dummy': ['__init__.pyi', 'py.typed']
}
)
From what I understand, the stub directory is treated as a separate package and installed alongside the extension. I've only tested it with vscode but I believe all IDEs should be able to parse it. Full code can be found here.
Related
My script, built as an application, stops working when it tries to load pickle model data. The following is the problematic code:
with open('model_pickle','rb') as f:
mp = pickle.load(f)
This is the setup file I’m using:
setup(
name="Test",
version="1.0",
options={"build_exe":{"include_files":["model_pickle"]}},
executables=[target]
)
The program works as it should when executing the python script. However, after converting it to an executable via cx_freeze, the issue is with opening the pickle model data file. I have tested this with and without the data, so I’m pretty sure this is the issue.
You can import model as follows model = pickle.load(open('MODEL_PATH','rb')) .Hope this will work!
I’m 99.99% positive that you need to also add the model_pickle file to your setup(…) call as package data for this to work as an executable. Here’s a setup.py snippet showing the package data options:
from setuptools import setup, find_packages
PROJECT_NAME = 'my_project' # this should reflect your package structure
setup(
# …
packages=[package for package in find_packages() \
if package.startswith(PROJECT_NAME)],
package_dir={ 'my_project' : 'my_project' },
package_data={ '' : ['*.*'] },
include_package_data=True,
zip_safe=True,
# …
)
Also, if I were you, I’d rename the data file to something like model_pickle.pkl so that your package_data expression doesn’t have to be double-wildcarded (as above).
If that doesn’t immediately work, I would also recommend adding a MANIFEST.in file that explicitly names your binary data file.
Have you tested your program with the IDLE ? Maybe there is a pickle decoding error... If not, try using this code for creating your exe. It imports all the libraries that you have on your computer. It's longer, but easier: just one file to run.
Anyways, better user PyInstaller (also a module available on PyPI) - Related Stack Overflow answer
I'm creating a C++ extension module for Python3. Compilation of the setup.py file compiles just fine, but when I go to import my new module, I get
ImportError: libMyLib.so: cannot open shared object file: No such file or directory
this is the path to my .so:
/path/to/lib-cc7/libMyLib.so
I've tried to import my libraries in the setup.py in different ways, I have tried setting and re-setting my LD_LIBRARY_PATH variable in the terminal as well as in my .bash_profile. I have tried setting the paths in sys.path.
When I run this code before the import statement:
print(os.environ.get("LD_LIBRARY_PATH"))
print(os.environ.get("PYTHONPATH"))
I get the path to the correct library directory.
When I run strace the path to other .so's that I need show up, and I see it searching for libMyLib.so, but it just searches in what seems like all of the other directories and /path/to/lib-cc7/. In other library searches it checks /path/to/lib-cc7/.
I have sanity checked that the library is there about 5 times.
It seems like no matter what I do,
import MyModule.MySubModule as SubModule
always returns the same import error. Is there anything else that I haven't tried? Why does it seem like Python is looking in the wrong place for my .so?
EDIT 1:
This is what my setup.py (in essence) looks like:
Submodule1 = Extension('Module.Submodule1', sources=['Module/Submodule1/submod1.cpp'], language = C++, libraries=[..long list of libraries..],)
Submodule2 = Extension('Module.Submodule2', sources=['Module/Submodule2/submod2.cpp'], language = C++, libraries=[..long list of libraries..],)
setup(name = "Module", version = '1.0',
packages = ['Module', 'Module.Submodule1', 'Module.Submodule2'],
ext_modules = [Submodule1, Submodule2], )
I have a python module fully implemented in python. (For portability reasons.)
The implementation of a small part has been duplicated in a cython module. To improve perfomance where possible.
I know how to install the .c modules created by cython with distutils. However if a machine has no compiler installed, I suspect the setup will fail even though the module is still usable in pure python mode.
Is there a way to compile the .c module if possible but fail gracefully and install without it if compiling is not possible?
I guess you will have to make some modification both in your setup.py and in one __init__ file in your module.
Let say the name of your package will be "module" and you have a functionality, sub for which you have pure python code in the sub subfolder and the equivalent C code in c_sub subfolder.
For example in your setup.py :
import logging
from setuptools.extension import Extension
from setuptools.command.build_ext import build_ext
from distutils.errors import CCompilerError, DistutilsExecError, DistutilsPlatformError
logging.basicConfig()
log = logging.getLogger(__file__)
ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError, IOError, SystemExit)
setup_args = {'name': 'module', 'license': 'BSD', 'author': 'xxx',
'packages': ['module', 'module.sub', 'module.c_sub'],
'cmdclass': {'build_ext': build_ext}
}
ext_modules = [Extension("module.c_sub._sub", ["module/c_sub/_sub.c"])]
try:
# try building with c code :
setup(ext_modules=ext_modules, **setup_args)
except ext_errors as ex:
log.warn(ex)
log.warn("The C extension could not be compiled")
## Retry to install the module without C extensions :
# Remove any previously defined build_ext command class.
if 'build_ext' in setup_args['cmdclass']:
del setup_args['cmdclass']['build_ext']
# If this new 'setup' call don't fail, the module
# will be successfully installed, without the C extension :
setup(**setup_args)
log.info("Plain-Python installation succeeded.")
Now you will need to include something like this in your __init__.py file (or at any place relevant in your case):
try:
from .c_sub import *
except ImportError:
from .sub import *
In this way the C version will be used if it was build, other-wise the plain python version is used. It assumes that sub and c_sub will provide the same API.
You can find an example of setup file doing this way in the Shapely package. Actually most of the code I posted was copied (the construct_build_ext function) or adapted (lines after) from this file.
Class Extension has parameter optional in constructor:
optional - specifies that a build failure in the extension should not
abort the build process, but simply skip the extension.
Here is also a link to the quite interesting history of piece of code proposed by mgc.
The question How should I structure a Python package that contains Cython code
is related, there the question is how to fallback from Cython to the "already generated C code". You could use a similar strategy to select which of the .py or the .pyx code to install.
I'm working with Python. My file GUI.py uses Yapsy for including new plugins into my software. I have finished all my work and now I want to use py2exe to generate a executable. I don't know how can I order py2exe to include Yapsy and all their modules. I have tried that and it doesn't work:
opts = {
'py2exe': { "includes" : ["yapsy.PluginManager","sip", "matplotlib.backends", "matplotlib.backends.backend_qt4agg",
"matplotlib.figure","pylab", "numpy","matplotlib.backends.backend_tkagg"]}
...
setup(
name='PImageQT',
version='1.0',
author='Jaime',
package={'images/*', 'Windows6S/*', 'modulos6S/*', 'OperacionesPlugins/*'},
scripts=['GUI.py'],
console=["GUI.py"],
options=opts,
data_files=data_files,
zipfile=None
)
I have found something in this link http://notinthestars.blogspot.com.es/2011/04/using-python-plugin-scripts-with-py2exe.html but it doesn't solve my problem.
In my Gui.py file for each plugin I add an action into a menu. This menu in the executable doesn't appear.
Does anyone know how I must do it?
Thank you very much.
Edit:
I have used breakpoints and yapsy works fine, but it doesn't found my plugins directory. The directory's name is the same.
Edit 2:
The path is ok and the directory is into the path. I don't know why it doesn't find the plugins directory after packing with py2exe. My code line for set the directory is the same like other web pages: self.manager.setPluginPlaces(["plugins"])
Edit 3:
I tried use a relative path to search the plugins and, it doesn't work.
self.manager.setPluginPlaces([os.getcwd() + os.sep + "plugins"])
Edit 4:
Today, I tried to list the directory were am I and my directory "plugins" is in. I don't understand nothing but the problem continue.
print(os.listdir('.'))
Edit 5:
This is the neverending story. I have used PYInstaller to make an executable file to check if the problem is py2exe. And it doesn't work. Now I think that the problem is Yapsy. Any ideas?
From your EDITS it seems that the problem is related to accessing some packaged plugin (and not the yapsy module per se).
For that you might want to check the following two points:
when you call self.manager.setPluginPlaces(["plugins"]) be aware that the path "plugin" may not be related to the directory you think. You should use a more specific path, using tips from the "where Am I" faq of py2exe: http://www.py2exe.org/index.cgi/WhereAmI
you might also try to package the plugins as data_files (like in the web page you've linked to), I'm not sure anymore how exactly the "package" argument (in your code sample) is handled by py2exe.
EDIT: for people that may come by later, the answer is in the comments below and basically what happened is that one of the plugin had an import error (a dependency not packaged by py2exe) and this import error could only be seen if python's standard logging is properly configured/activated with:
import logging
logging.basicConfig(level=logging.DEBUG)
you probably need to include all yapsy
opts = {
'py2exe': { "includes" : ["yapsy","sip", "matplotlib.backends", "matplotlib.backends.backend_qt4agg",
"matplotlib.figure","pylab", "numpy","matplotlib.backends.backend_tkagg"]}
...
setup(
name='PImageQT',
version='1.0',
author='Jaime',
package={'images/*', 'Windows6S/*', 'modulos6S/*', 'OperacionesPlugins/*'},
scripts=['GUI.py'],
console=["GUI.py"],
options=opts,
data_files=data_files,
zipfile=None
)
just a wild guess however
I have a problem with making exe using py2exe. In my project i'm using sqlalchemy with mssql module.
My setup.py script looks like:
from distutils.core import setup
import py2exe
setup(
windows=[{"script" : "pyrmsutil.py"}],
options={"pyrmsutil" : {
"includes": ["sqlalchemy.dialects.mssql", "sqlalchemy"],
"packages": ["sqlalchemy.databases.mssql", "sqlalchemy.cresultproxy"]
}})
But when i'm starting procedure like:
python.exe setup.py py2exe
I'm receiving build log with following errors:
The following modules appear to be missing
['_scproxy', 'pkg_resources', 'sqlalchemy.cprocessors', 'sqlalchemy.cresultproxy']
And in "dist" folder i see my pyrmsutil.exe file, but when i'm running it nothing happens. I mean that executable file starts, but do nothing and ends immediately without any pyrmsutil.exe.log. It's very strange.
Can anybody help me with this error?
I know it's no an answer per se but have you tries pyInstaller? I used to use py2exe and found it tricky to get something truly distributable. pyInstaller requires a little more setup but the docs are good and the result seems better.
For solving this issue you could try searching for the mentioned dlls and placing them in the folder with the exe, or where you build it.
Looks like py2exe can't find sqlalchemy c extensions.
Why not just include the egg in the distribution, put sqlachemy in py2exe's excludes and load the egg on start?
I use this in the start script:
import sys
import path
import pkg_resources
APP_HOME = path.path(sys.executable).parent
SUPPORT = APP_HOME / 'support'
eggs = [egg for egg in SUPPORT.files('*.egg')]
reqs, errs = pkg_resources.working_set.find_plugins(
pkg_resources.Environment(eggs)
)
map(pkg_resources.working_set.add, reqs)
sys.path.extend(SUPPORT.files('*.egg'))
i use Jason Orendorff's path module (http://pypi.python.org/pypi/path.py) but you can easily wipe it out if you want.