We Keep Coding

numpy.f2py - extension doesn't exist when source distribution is installed with pip

I'm trying to wrap my head around f2py because my organization has a lot of legacy fortran code that I would like to incorporate into some newer python-based tools I'm writing. Ideally, I would package these tools either in source packages or wheels to make it easier to distribute to the rest of the organization.
I've written a small test package based on some other examples I've seen that just sums an array of floats. The package contents are included below. If I build a source distribution tarball using py setup.py sdist, everything looks like it works. It even looks like pip successfully installs it. However, if I open a python shell and try to import the newly installed module, I get an error on the from fastadd import fadd line in the initialization script saying
AttributeError: module 'fastadd' has no attribute 'fastadd'
So it seems like it didn't actually successfully build the f2py module. Doing some troubleshooting, if I open a powershell window in the package folder and just run
py -m numpy.f2py -c fadd.pyf fadd.f90
and then open a python shell in the same folder and try to import fastadd, I get an error, ImportError: DLL load failed: The specified module could not be found. (This is after I installed the Visual Studio build tools, a fix suggested on several threads). Following the advice on this thread, changing the command to
py -m numpy.f2py -c --fcompiler=gnu95 --compiler=mingw32 fadd.pyf fadd.f90
will build a module file that I can successfully import and use. Okay, great.
However, when I change config.add_extension in the setup file to include the keyword argument f2py_options=["--fcompiler=gnu95","--compiler=mingw32"] and try to build a package distribution file with setup.py sdist command and then install using py -m pip install fastadd-1.0a1.tar.gz, I get yet a different error that says
ERROR: No .egg-info directory found in C:\Users\username\AppData\Local\Temp\pip-pip-egg-info-c7406k03
And now I'm completely flummoxed. Other configurations of the f2py_options either result in setup.py throwing an error or fail to create the extension altogether, similar to above. Using a simple string for the options gives an error, so apparently f2py_options does in fact expect a list input. I can't seem to find any good documentation on whether I'm using f2py_options correctly, and I have no idea why just adding that option would cause pip to not know where its info directory is. That makes no sense to me. I'd really appreciate some help on this one.
I'm running Python 3.7.0 32-bit, numpy 1.20.1, and pip 21.0.1 on a Windows 10 machine.
--EDIT--
Looking in the installation directory of the test module, I found a new wrinkle to this problem: the installation directory does not actually include any files listed in MANIFEST, not even the __init__.py file. If I copy __init__.py into the directory, trying to import the module gives the same ImportError: DLL load failed error I've been getting.
Also, inspecting the output of py -m pip install, it looks like numpy.distutils doesn't recognize --fcompiler or --compiler as valid options and just ignores them, even though numpy.f2py does recognize them.
--END EDIT--
PACKAGE CONTENTS:
+-fastadd
---__init__.py
---fadd.f90
---fadd.pyf
-MANIFEST.in
-README
-setup.py
fadd.f90 has the following contents:
subroutine fadd(vals,n,mysum)
integer, intent(in) :: n
real*8, intent(out):: mysum
real*8, dimension(n), intent(in) :: vals
mysum = sum(vals)
end subroutine fadd
fadd.pyf has the following contents:
python module fastadd ! in
interface ! in :fastadd
subroutine fadd(vals,n,mysum) ! in :fastadd:fadd.f90
real*8 dimension(n),intent(in) :: vals
integer, optional,intent(in),check(len(vals)>=n),depend(vals) :: n=len(vals)
real*8 intent(out) :: mysum
end subroutine fadd
end interface
end python module fastadd
__init__.py:
"""This is the documentation!"""
from .fastadd import fadd
MANIFEST.in:
include README
recursive-include fastadd *.f90
recursive-include fastadd *.pyf
recursive-include fastadd *.py
and, finally, setup.py:
def configuration(pth=None):
from numpy.distutils.misc_util import Configuration
config = Configuration(
'fastadd',
top_path=pth,
version='1.0a1',
author='John Doe',
author_email='john.doe#fake-org.biz',
url='fake-org.biz/fastadd',
description="Testing f2py build process. Sums an arbitrary-length list of numbers.")
config.add_extension(
'fastadd',
sources=['fastadd\\fadd.pyf','fastadd\\fadd.f90']
)
return config
if __name__ == '__main__':
from numpy.distutils.core import setup
setup(**configuration('fastadd').todict())
If it helps at all, the final MANIFEST file looks like this after the setup script is run:
# file GENERATED by distutils, do NOT edit
README
setup.py
C:\Users\username\Documents\Development\python_modules\fastadd\fastadd\fadd.f90
C:\Users\username\Documents\Development\python_modules\fastadd\fastadd\fadd.pyf
fastadd\__init__.py
fastadd\fadd.f90
fastadd\fadd.pyf

How to "hide" some .py files into .exe with cx_Freeze?

Env:
Windows 10
python 3.6.6
cx-Freeze 5.0.2
Git hub example
It contails .msi for installing
Example project structure:
/package_name
/some_packet
/__init.py
/module_name.py # for an example contains valiable in code "module_1"
/main.py
/setup.py
/some_module.py # for an example contains valiable "module_2"
/some_other_module.py # for an example contains valiable "module_3"
Example of setup.py(simplified)
import cx_Freeze
cx_Freeze.setup(
name="example",
options={
"build_exe": {
"packages": ["asyncio"],
"include_files": ["static\some_static_file.png"]
},
"bdist_msi": {
"upgrade_code": "{492de237-1853-4599-a707-c283d567699f}"
}
},
executables=[cx_Freeze.Executable("main.py")]
)
Current behavior
For creating .msi install file -> run command python setup.py bdist_msi. It will generate .msi files for installing application.
After installing this application: directory(where application is installed) will contain:
main.exe
lib\some_packet directory
lib\some_packet\module_name.pyc file
other files
There are following statements:
1) From root directory(where application is installed) i start search(via grep -Rna command under Ubuntu guest system, it's just more convenient for me) and valiable module_1 could be found in directories(in lib\some_packet\module_name.pyc) and module_2/module_3 couldn't be found. Details:
(v_decompile) any#any-pc:/mnt/hgfs/shar/example_installed$ grep -Rna "module_1"
lib/some_packet/module_name.pyc:2:B�!]�#dZdS)module_1N)r�rr�PG:\heroes\installer-with-cx_Freeze\sources_of_project\some_packet\module_name.py<module>s
(v_decompile) any#any-pc:/mnt/hgfs/shar/example_installed$ grep -Rna -a "module_2"
(v_decompile) any#any-pc:/mnt/hgfs/shar/example_installed$ grep -Rna -a "module_3"
2) File lib\some_packet\module_name.pyc could be easily converted to original file(without comments) by e.g. python-uncompyle6.
Details:
(v_decompile) any#any-pc:/mnt/hgfs/shar/example_installed$ uncompyle6 lib/some_packet/module_name.pyc
# uncompyle6 version 3.3.3
# Python bytecode 3.6 (3379)
# Decompiled from: Python 3.6.6 (default, Jul 20 2018, 15:39:05)
# [GCC 4.8.4]
# Embedded file name: G:\heroes\installer-with-cx_Freeze\sources_of_project\some_packet\module_name.py
# Compiled at: 2019-07-07 11:28:50
module_1 = 'module_1'
# okay decompiling lib/some_packet/module_name.pyc
3) (solved with this question) In both points: file contains source path G:\heroes\installer-with-cx_Freeze\sources_of_project\some_packet\module_name.py It confuses me a bit. Application was installed from .msi and (as I understand) should not know about source directories (regarding path) which was used for creating last one.
Questions:
Is there any way to recover some_module.py and some_other_module.py to original files from main.exe?(like it could be done with lib\some_packet\module_name.pyc)
How to "hide" some other files in application into main.exe or somehow avoid converting .pyc to original files.(maybe some properties in cx_Freeze?)
Note:
It should be done with cx_Freeze.
PS: I don't want to create single .exe. I try to find convenient way for specifying which files should be stored in main.exe like it was done with some_module.py and some_other_module.py
PSS: At this moment I see only I way: put all files on main.py level :) But it will look weird for big project.

Quoting How to obfuscate Python source code:
These two methods [using pyobfuscate and distributing bytecode] are really just a deterrent, not a secure way of hiding the code.
If you want something a bit more robust, you should take a look at Nuitka, which compiles Python code to C++, so you can compile that and just distribute the executable. It seems to be broadly compatible with different libraries and different versions of Python.
See also Python Code Obfuscation

This video might help.
It talks about cxfreeze and how you can use cxfreeze to make a excutable, and I know it works for 3.4+ because the video uses python 3.4, but really your method should be fine...

If I understand Your question correctly, this could be answer You want - Python cx_Freeze for two or more python files (modules)

Where are Python wheels created ?

I do this
C:\WINDOWS\system32>python c:\\python34\kissdownloader\setup.py bdist_wheel
and I get this after some other stuff
creating build\bdist.win-amd64\wheel\kissdownloader-1.dist-info\WHEEL
I goto my python root and there's nothing there , no build folder or anything . Is the thing somewhere else ? Oh and I don't have a manifest.in file if that makes any difference .
My setup.py file looks like this :
from setuptools import setup
setup (
name='kissdownloader',
version='1',
description = 'A kisscartoon/kissanime downloader' ,
package_dir={'kissdownloader':'C:\\Python34'},
author='Vriska',
author_email='xyz#gmail.com',
install_requires = ['bs4','cfscrape','requests'],
package_data={'data' : ['C:\Program Files\PhantomJS\phantomjs.exe']},
)

The output is generated in your current working directory. In your case, that's C:\WINDOWS\system32, because that's where you started Python. You'll find a build and dist directory; the latter contains the completed wheel.
If you want the directories to be created next to setup.py, change your directory to c:\python34\kissdownloader\ first. Many a Python project expects you to run setup.py there anyway.
As a side note: I wouldn't bundle the PhantomJS binary in with your project. Instead, require users to install it separately; they may already have it installed anyway, and you may run into legal issues by re-distributing it in your own project without at least a compatible license.

Resolving Python dependencies at installation time in setup.py based on an interpreter version

Suppose I have a bunch of requirement files like:
requirements.txt # common for both 2.x and 3.x
requirements-2.txt # 2.x
requirements-3.txt # 3.x
and I would like to populate install_requires argument in setup.py file based on the current Python interpreter version. Assume of course that pip handles the installation process.
Solution 1: Of course, I can write a simple function that will read and return correct requirements. In a case with multiple projects this is obviously not acceptable, since I will have to copy the function everywhere.
Solution 2: Next idea here is to write a simple package that does it for me, but the problem is that it should be available not only at distribution time (like python setup.py sdist), but more importantly, at installation time on one's machine.
I was managed to write a simple module that does the thing, lets call it depmodule. I also had following setup.py:
# -*- coding: utf-8 -*-
from setuptools import setup, find_packages
try:
from depmodule import find_requirements
except ImportError:
# this line is executed when reading setup.py for the first time
# since depmodule is not installed yet
find_requirements = lambda: []
setup(
name='some-package',
packages=find_packages(),
# snip...
platforms='any',
# note that depmodule is listed here as a requirement, so it will be
# installed before some-package, thus will be available when it comes
# to running setup.py of some-package
install_requires=['depmodule'] + find_requirements(),
)
When it comes to pip install some-package it actually resolves dependencies correctly, but they are not picked up by pip, so it only installs: depmodule some-package (in that order) instead of depmodule dep1 dep2 ... some-package.
I tried to use setup_requires argument but with no luck. The dependency was downloaded, but I could not access it, since it was an egg package (not extracted).
Is there any way I can overcome this issue? Are there any alternatives (other approaches) that could help with this?
Thanks!

Since you do not want to replicate code for multiple projects you might consider generating the setup.py and/or updating some part of setup.py from from a single source.
This would be similar to generate a Makefile from a Makefile.in template.
I have a py_setup program for this. I call it with py_setup --new to generate a new setup.py in the current directory, taking parts of py_setup as a template. When py_setup is
run with a filename as argument, it tries to update segments in that file but leaves the rest of them untouched.
When run without arguments and without options py_setup does update segments on all */setup.py files.
The segments in py_setup (and the setup.py files) are seperated by comment lines of the form #_## segement_name or ended with #_#. Anything before or between segments is copied, but will never get updated in the setup.py
Anything after the line #_### in the py_setup is never copied, that is the actual code that is the actual py_setup program code
Most lines are copied verbatim, except for the segment separator comments (stripped after the segment name) and the line that starts with:
setup = setup
From that line setup = is stripped at the start, so it ends up as a call to setup() in the setup.py but not when running py_setup.
When updating, only existing segments that are in the target setup.py are replaced with lines from the same named segments in py_setup. Removing or changing the segment name makes sure code changes will not be udpated.

Python: How to include the exe file of a script in setup.py

I am trying to learn Python by myself using Zed A.Shaw's book Learn Python the hard way.
At exercise 46. I'am supposed to create a project skeleton (i.e. create a setup.py file, create modules, and so). Then make a project.
I have to put a script in my bin directory that is runnable for my system. I wrote the simple Hello World! script turned it into an .exe file using cxfreeze.
However when I try to install my setup.py file (i.e. By typing python setup.py install in the cmd), I can't install this .exe file instead I can only install the script script.py
How can I install this exe file.
This is my setup.py file:
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
config = {
'description': 'First project',#ex46
'author': 'author',#
'url': '',#N/A
'download_url': '',#N/A
"author_email": "author_email#email.com"
'versio': '3.1',
'install_requires': ['nose'],
'packages': ['skeleton\quiz46','skeleton\\tests'],
'scripts': ['skeleton\\bin\helloscript.py','skeleton\\bin\helloscript.exe'],
'name': 'quiz46'
}
But this gives me the following error:
UnicodeDecodeError
I have also tried putting skeleton\bin\helloscript.exe but that gives me a similiar Error!
My OS is Windows 7, and I am using Python 3.1.
Again what I want is for the setup.py to install my .exe file too not just it's script.

I don't think the script option is meant to handle anything but text files. If you have a look at the source code for distribute (aka setuptools), the write_script command will try to encode('ascii') the contents if it's anything other than a python script AND if you are using Python 3. Your cxfreeze exe is a binary file, not a text file, and is likely causing this to choke.
The easier option to get setuptools to include a executable script in the installation process is to use the entry_points option in your setup.py rather than scripts:
entry_points={'console_scripts':['helloscript = helloscript:main'] }
The console_script will automatically wrap your original helloscript.py script and create an exe (on Windows) and install it into your Python's Script directory. No need to use something like cxfreeze.