We Keep Coding

Pyinstaller to package app hidden-import warning/exceptions

So, I've had some trouble setting up my application through Inno setup Compiler, at first I assumed it was a problem within Inno itself but on further inspection I think it is my actual exe. I am able to create a working exe file that runs my program properly but only on my own pc. I am able to create the setup file that also works through Inno setup but it only works on my own pc. I have sent both the actual exe file and the Inno setup file to another computer and downloaded it there and ran it, both meet the same "Fatal Error: failed to run script tk_app.py". Therefore, the problem must be that I have not been able to pavkage the app properly with pyinstaller.
My prgoram has 5 files in total (all in the same folder): main.py, file1.py, file2.py file3.py, tk_app.py
All of them importing each other and using python libraries. I know that pyinstaller supports librarires such as pandas, tkinter and many more without needing the --hidden-impoort= command and that it will pick up all files within the program if there are files that are importing each other.
So I use the file tk_app.py (which contains my tkinter UI and imports main.py which then goes onto import file1.py which import another file so on)
The pyinstaller command line I use to make the exe is as follows:
PS C:\Users\ripta\Desktop\CODING\CSV_Upload> pyinstaller -w --hidden-import=bs4 --hidden-import=fake_useragent --hidden-import=urllib.prase --hidden-import=urllib.request --hidden-import=os --hidden-import=pandas.io.parsers --icon=trademark_icon.ico --onefile tk_app.py
My question is, will pyinstaller tell me when it needs a given --hidden-import='name' when running becuase it doesn not throw up any errors and does produce a spec file, a build folder and a dist folder containing the exe file.
I have noticed that it throws up WARNINGs or Exceptions (also not sure why it mentions django as I do not import or use it at all inthe application) :
59182 INFO: Loading module hook 'hook-django.db.backends.py' from 'c:\\users\\ripta\\appdata\\local\\programs\\python\\python36-32\\lib\\site-packages\\PyInstaller\\hooks'... 61711 WARNING: Hidden import "django.db.backends.__pycache__.base" not found! 61712 INFO: Loading module hook 'hook-django.py' from 'c:\\users\\ripta\\appdata\\local\\programs\\python\\python36-32\\lib\\site-packages\\PyInstaller\\hooks'... Traceback (most recent call last): File "<string>", line 21, in walk_packages File "c:\users\ripta\appdata\local\programs\python\python36-32\lib\site-packages\django\contrib\gis\admin\__init__.py", line 5, in <module>
Or show Hidden-import not found (Of imports I have no idea about):
149329 WARNING: Hidden import "pkg_resources.py2_warn" not found! 149330 WARNING: Hidden import "pkg_resources.markers" not found!
The fact that the script fails to run on any other computer besided my own leads me to think that it must be lacking a dependency that is only found on my computer, therefore I am not using pyinstaller correctly but am not too sure where exactly I am making the mistake.

First I've uninstalled Setuptools and reinstalled it with specific version. Then, I've imported pkg_resources.py2_warn as hidden import :
pip uninstalled setuptools
pip install --upgrade 'setuptools<45.0.0'
pyinstaller --hidden-import='pkg_resources.py2_warn' tk_app.py
It worked for me.

numpy.f2py - extension doesn't exist when source distribution is installed with pip

I'm trying to wrap my head around f2py because my organization has a lot of legacy fortran code that I would like to incorporate into some newer python-based tools I'm writing. Ideally, I would package these tools either in source packages or wheels to make it easier to distribute to the rest of the organization.
I've written a small test package based on some other examples I've seen that just sums an array of floats. The package contents are included below. If I build a source distribution tarball using py setup.py sdist, everything looks like it works. It even looks like pip successfully installs it. However, if I open a python shell and try to import the newly installed module, I get an error on the from fastadd import fadd line in the initialization script saying
AttributeError: module 'fastadd' has no attribute 'fastadd'
So it seems like it didn't actually successfully build the f2py module. Doing some troubleshooting, if I open a powershell window in the package folder and just run
py -m numpy.f2py -c fadd.pyf fadd.f90
and then open a python shell in the same folder and try to import fastadd, I get an error, ImportError: DLL load failed: The specified module could not be found. (This is after I installed the Visual Studio build tools, a fix suggested on several threads). Following the advice on this thread, changing the command to
py -m numpy.f2py -c --fcompiler=gnu95 --compiler=mingw32 fadd.pyf fadd.f90
will build a module file that I can successfully import and use. Okay, great.
However, when I change config.add_extension in the setup file to include the keyword argument f2py_options=["--fcompiler=gnu95","--compiler=mingw32"] and try to build a package distribution file with setup.py sdist command and then install using py -m pip install fastadd-1.0a1.tar.gz, I get yet a different error that says
ERROR: No .egg-info directory found in C:\Users\username\AppData\Local\Temp\pip-pip-egg-info-c7406k03
And now I'm completely flummoxed. Other configurations of the f2py_options either result in setup.py throwing an error or fail to create the extension altogether, similar to above. Using a simple string for the options gives an error, so apparently f2py_options does in fact expect a list input. I can't seem to find any good documentation on whether I'm using f2py_options correctly, and I have no idea why just adding that option would cause pip to not know where its info directory is. That makes no sense to me. I'd really appreciate some help on this one.
I'm running Python 3.7.0 32-bit, numpy 1.20.1, and pip 21.0.1 on a Windows 10 machine.
--EDIT--
Looking in the installation directory of the test module, I found a new wrinkle to this problem: the installation directory does not actually include any files listed in MANIFEST, not even the __init__.py file. If I copy __init__.py into the directory, trying to import the module gives the same ImportError: DLL load failed error I've been getting.
Also, inspecting the output of py -m pip install, it looks like numpy.distutils doesn't recognize --fcompiler or --compiler as valid options and just ignores them, even though numpy.f2py does recognize them.
--END EDIT--
PACKAGE CONTENTS:
+-fastadd
---__init__.py
---fadd.f90
---fadd.pyf
-MANIFEST.in
-README
-setup.py
fadd.f90 has the following contents:
subroutine fadd(vals,n,mysum)
integer, intent(in) :: n
real*8, intent(out):: mysum
real*8, dimension(n), intent(in) :: vals
mysum = sum(vals)
end subroutine fadd
fadd.pyf has the following contents:
python module fastadd ! in
interface ! in :fastadd
subroutine fadd(vals,n,mysum) ! in :fastadd:fadd.f90
real*8 dimension(n),intent(in) :: vals
integer, optional,intent(in),check(len(vals)>=n),depend(vals) :: n=len(vals)
real*8 intent(out) :: mysum
end subroutine fadd
end interface
end python module fastadd
__init__.py:
"""This is the documentation!"""
from .fastadd import fadd
MANIFEST.in:
include README
recursive-include fastadd *.f90
recursive-include fastadd *.pyf
recursive-include fastadd *.py
and, finally, setup.py:
def configuration(pth=None):
from numpy.distutils.misc_util import Configuration
config = Configuration(
'fastadd',
top_path=pth,
version='1.0a1',
author='John Doe',
author_email='john.doe#fake-org.biz',
url='fake-org.biz/fastadd',
description="Testing f2py build process. Sums an arbitrary-length list of numbers.")
config.add_extension(
'fastadd',
sources=['fastadd\\fadd.pyf','fastadd\\fadd.f90']
)
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(**configuration('fastadd').todict())
If it helps at all, the final MANIFEST file looks like this after the setup script is run:
# file GENERATED by distutils, do NOT edit
README
setup.py
C:\Users\username\Documents\Development\python_modules\fastadd\fastadd\fadd.f90
C:\Users\username\Documents\Development\python_modules\fastadd\fastadd\fadd.pyf
fastadd\__init__.py
fastadd\fadd.f90
fastadd\fadd.pyf

Python and C++ (Pybind11): Python Extension Module Import Error

I try to wrap a C++ - function with pybind to create a python-module. After compiling with Visual Studio I accomplished to create a .pyd file.
I tried to import this file (python>>> import pymodule) but get this error:
Import Error: dynamic module does not define module export function (PyInit_pymodule)
Is it usually the right approach to import a .pyd file the way I explained above? Why do I get this error?

1a) You have to copy the .pyd file into the Python-installation folder (Lib-> side packages).
1b) You can open Python in the same folder where the .pyd - file is placed.
2) the python module must have the same name as the directory defined in CMake

Compiling an optional cython extension only when possible in setup.py

I have a python module fully implemented in python. (For portability reasons.)
The implementation of a small part has been duplicated in a cython module. To improve perfomance where possible.
I know how to install the .c modules created by cython with distutils. However if a machine has no compiler installed, I suspect the setup will fail even though the module is still usable in pure python mode.
Is there a way to compile the .c module if possible but fail gracefully and install without it if compiling is not possible?

I guess you will have to make some modification both in your setup.py and in one __init__ file in your module.
Let say the name of your package will be "module" and you have a functionality, sub for which you have pure python code in the sub subfolder and the equivalent C code in c_sub subfolder.
For example in your setup.py :
import logging
from setuptools.extension import Extension
from setuptools.command.build_ext import build_ext
from distutils.errors import CCompilerError, DistutilsExecError, DistutilsPlatformError
logging.basicConfig()
log = logging.getLogger(__file__)
ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError, IOError, SystemExit)
setup_args = {'name': 'module', 'license': 'BSD', 'author': 'xxx',
'packages': ['module', 'module.sub', 'module.c_sub'],
'cmdclass': {'build_ext': build_ext}
}
ext_modules = [Extension("module.c_sub._sub", ["module/c_sub/_sub.c"])]
try:
# try building with c code :
setup(ext_modules=ext_modules, **setup_args)
except ext_errors as ex:
log.warn(ex)
log.warn("The C extension could not be compiled")
## Retry to install the module without C extensions :
# Remove any previously defined build_ext command class.
if 'build_ext' in setup_args['cmdclass']:
del setup_args['cmdclass']['build_ext']
# If this new 'setup' call don't fail, the module
# will be successfully installed, without the C extension :
setup(**setup_args)
log.info("Plain-Python installation succeeded.")
Now you will need to include something like this in your __init__.py file (or at any place relevant in your case):
try:
from .c_sub import *
except ImportError:
from .sub import *
In this way the C version will be used if it was build, other-wise the plain python version is used. It assumes that sub and c_sub will provide the same API.
You can find an example of setup file doing this way in the Shapely package. Actually most of the code I posted was copied (the construct_build_ext function) or adapted (lines after) from this file.

Class Extension has parameter optional in constructor:
optional - specifies that a build failure in the extension should not
abort the build process, but simply skip the extension.
Here is also a link to the quite interesting history of piece of code proposed by mgc.

The question How should I structure a Python package that contains Cython code
is related, there the question is how to fallback from Cython to the "already generated C code". You could use a similar strategy to select which of the .py or the .pyx code to install.

python + distutils: add .py file to package at build/install time?

For various not-very-good-but-unfortunately-necessary reasons I'm using a setup.py file to manage some binary assets.
During py setup.py build or install I would like to create a .py file in the "normal" Python package being installed by setup.py which contains some details about these binary assets (their absolute path, version information, etc).
What's the best way to create that file?
For example, I'd like it to work something like this:
$ cd my-python-package
$ py setup.py install
...
Installing version 1.23 of my_binary_assets to /some/path...
...
$ python -c "from my_python_package import binary_asset_version_info as info; print info"
{"path": "/some/path", "version": "1.23"}
(note: I'm using the cmdclass argument to setup(…) to manage the building + installation of the binary assets… I'd just like to know how to create the binary_asset_version_info.py file used in the example)

At first sight, there is a catch-22 in your requirements: The most obvious place to create this .py file would be in the build or build_py command (to get usual distutils operations like byte-compilation), but you want that file to contain the paths to the installed assets, so you’d have to create it during the install step. I see two ways to solve that:
a) Create your info.py file during build_py, and use distutils machinery to get the installation paths of the assets files
b) Create info.py during install and call distutils.util.byte_compile to byte-compile it
I find both ideas distasteful, but well :) Now, do you know how to fill in the file (i.e. get the install paths from distutils)?