I am creating a python package with a c++ extension. I am trying to do this with setuptools as that seems to be the preferred solution. Given that it allows for setup_requires, install_requires, I agree and use it for my python only packages. However, I am unable to get it to work when I have a c++ extension module. Then I resort to distutils.core to get it to work. I would like to know how to get it to work using setuptools. My setup script looks like
from setuptools import setup
import shutil
import os
# folder where .so is being build by cpp compilation
so_src = os.path.join(dir, 'cpp/build/')
# folder where .so should live in python package
so_des = os.path.join(dir, 'package_py/cpp/')
# extension module
lib_files = ['cpp_py.so']
# copy shared lib
for f in lib_files:
shutil.copyfile(so_src + f, so_des + f)
# set-up script
setup(
name=DISTNAME
, version=FULLVERSION
, description= DESCRIPTION
, author= AUTHOR
, author_email= EMAIL
, maintainer= AUTHOR
, maintainer_email= EMAIL
, long_description=LONG_DESCRIPTION
, packages=['package_py',
'package_py.cpp']
, package_dir={'package_py.cpp': 'package_py/cpp'}
, package_data={'package_py.cpp': lib_files}
)
This results in a package_py-0.0.1-py3.6.egg file in my python site-packages. The package only works when used from the installation folder.
Changing the first line to use distutils.core instead of setuptools
from distutils.core import setup # only change, remainder is the same!!
import shutil
import os
# folder where .so is being build by cpp compilation
so_src = os.path.join(dir, 'cpp/build/')
# folder where .so should live in python package
so_des = os.path.join(dir, 'package_py/cpp/')
# extension module
lib_files = ['cpp_py.so']
# copy shared lib
for f in lib_files:
shutil.copyfile(so_src + f, so_des + f)
# set-up script
setup(
name=DISTNAME
, version=FULLVERSION
, description= DESCRIPTION
, author= AUTHOR
, author_email= EMAIL
, maintainer= AUTHOR
, maintainer_email= EMAIL
, long_description=LONG_DESCRIPTION
, packages=['package_py',
'package_py.cpp']
, package_dir={'package_py.cpp': 'package_py/cpp'}
, package_data={'package_py.cpp': lib_files}
)
I get a package_py folder (with .so file) and package_py-0.0.1-py3.6.egg-info in site-packages. Now the module works in all folders.
As I would like to extend the python package to also use setup_requires, instal_requires I want to really use setuptools. How can I get the package to work in all folders using setuptools instead of distutils.core
If you're using a C extension, you should use Extension from either setuptools or distutils as well, it's not a python package, so I'm surprised it's even managing to install. If you plan on distributing the package to others, you shouldn't compile the extension beforehand, but have it compiled during the package installation so that the user's system compiles it appropriately (i.e., an .so file doesn't help a Windows user who needs it compiled into a .dll file, etc).
Try something like this:
from setuptools import setup, Extension
import shutil
import os
# I don't know how you want to build your extension or your file structure,
# so removing the build stuff.
your_modulename=Extension('_extensionname',
sources=['path/to/extension.cpp', 'more/file/paths'],
language='c'
)
# set-up script
setup(
name=DISTNAME
, version=FULLVERSION
, description= DESCRIPTION
, author= AUTHOR
, author_email= EMAIL
, maintainer= AUTHOR
, maintainer_email= EMAIL
, long_description=LONG_DESCRIPTION
, ext_modules=[your_modulename]
, packages=['package_py']
)
Hope this helps or at least gets you on the right track.
Related
I have a compiled a Python extension.
The resulting binary mylib.so file can be imported in the Python script and works fine.
Now I am wondering how to write the interface stub file mylib.pyi such, that pylint and the Python language server used in VS Code can use it?
For now the native library is just in the root of my scripts.
When putting the mylib.pyi next to it pylint ignores it.
I know this question is a year old but the answer might help the next person. I managed to do this by writing the following setup file:
from distutils.core import setup, Extension
extension = Extension(
name = 'dummy',
sources = ...,
...
)
setup(
name = 'dummy',
ext_modules = [extension],
packages = ['dummy'],
package_dir = {'dummy' : './stub'},
package_data = {
'dummy': ['__init__.pyi', 'py.typed']
}
)
From what I understand, the stub directory is treated as a separate package and installed alongside the extension. I've only tested it with vscode but I believe all IDEs should be able to parse it. Full code can be found here.
I am trying to generate aa .exe using py2exe for a python script that generates an excel. Here is just a sample code. I am writing value 100 to a cell and saving the excel to Users Desktop using openpyxl. This works perfectly fine when I run it directly.
import openpyxl
import getpass
wb = openpyxl.Workbook()
ws = wb.create_sheet('test')
ws.cell(row=1, column=1, value=100)
username = getpass.getuser()
wb.save('C:\\Users\\{}\\create_exe\\gen.xlsx'.format(username))
print 'Done'
And when I compile it using py2exe it compiles also just fine.
The problem arises when I run the generated .exe file. I get a return saying
ImportError: No module named jdcal
setup.py file is as follows
import py2exe
from distutils.core import setup
packages = ["openpyxl", "openpyxl.workbook", "xml.etree", "xml"]
excludes = []
setup(console=['test_program.py'],
options={"py2exe": {"excludes": excludes,
"packages": packages}}
)
Thisngs I have already tried
I have searched and few people said Install openpyxl using pip. I
have done that and pip says its alöready installed.
I have also tried to install jdcal using pip and pip says it is installed.
I have uninstalled jdcal and installed it using pip and manually, and
still the same error.
I have included jdcal in the packages and still no change in the outcome.
I hope someone can help me with it.
Thanks in advance
EDIT :
Generated Filed in dist folder are as follows (openpyxl cannot be seen here, I don't know why)
tcl (Folder)
_ctypes.pyd
_elementtree.pyd
_hashlib.pyd
_multiprocessing.pyd
_socket.pyd
_ssl.pyd
_tkinter.pyd
bz2.pyd
pyexpat.pyd
select.pyd
unicodedata.pyd
win32ui.pyd
numpy.core._dummy.pyd
numpy.core.multiarray.pyd
numpy.core.multiarray_tests.pyd
numpy.core.operand_flag_tests.pyd
numpy.core.struct_ufunc_test.pyd
numpy.core.test_rational.pyd
numpy.core.umath.pyd
numpy.core.umath_tests.pyd
numpy.fft.fftpack_lite.pyd
numpy.linalg._umath_linalg.pyd
numpy.linalg.lapack_lite.pyd
numpy.random.mtrand.pyd
_win32sysloader.pyd
win32api.pyd
win32pdh.pyd
win32pipe.pyd
tk85.dll
tcl85.dll
libiomp5md.dll
pywintypes27.dll
python27.dll
w9xpopen.exe
pythoncom27.dll
library.zip
test_program.exe (Executable File)
Try to manually include it in setup.py in packages = ["openpyxl", "openpyxl.workbook", "xml.etree", "xml"]
so it would be:
packages = ["openpyxl", "openpyxl.workbook", "xml.etree", "xml", "jdcal"]
I personally have had good luck letting py2exe detect the modules that are needed. I've never tried to specify every module necessary.
try this:
from distutils.core import setup
import py2exe
setup(console=['test_program.py'])
this should be run from command line as
python setup.py py2exe
py2exe outputs .dll files in the dist directory, these have to be in the directory that you run your .exe file from. if you just want one .exe file and no .dll files try this:
from distutils.core import setup
import py2exe, sys, os
sys.argv.append('py2exe')
setup(
options = {'py2exe': {'bundle_files': 1, 'compressed': True}},
console = [{'script': "test_program.py"}],
zipfile = None,
)
this should be run from command line as
python setup.py
I use cx_freeze and never had any issues.
Here's the setup.py for a cx_freeze file
from cx_Freeze import setup, Executable
build_exe_options = {"excludes": ["html5lib"],"optimize":2}
setup(name = "App Name" ,
version = "1.0.0.0" ,
options = {"build_exe": build_exe_options},
description = "" ,
executables = [Executable("FooBar.py")])
I am making a Python package that has a C++-extension module and someone else's shared library that it requires. I want everything installable via pip. My current setup.py file works when I use pip install -e . but when I don't use develop mode (e.i. omit the -e) I get "cannot open shared object file" when importing the module in Python. I believe the reason is that setuptools doesn't consider the shared library to be part of my package, so the relative link to the library is broken during installation when files are copied to the install directory.
Here is what my setup.py file looks like:
from setuptools import setup, Extension, Command
import setuptools.command.develop
import setuptools.command.build_ext
import setuptools.command.install
import distutils.command.build
import subprocess
import sys
import os
# This function downloads and builds the shared-library
def run_clib_install_script():
build_clib_cmd = ['bash', 'clib_install.sh']
if subprocess.call(build_clib_cmd) != 0:
sys.exit("Failed to build C++ dependencies")
# I make a new command that will build the shared-library
class build_clib(Command):
user_options = []
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
run_clib_install_script()
# I subclass install so that it will call my new command
class install(setuptools.command.install.install):
def run(self):
self.run_command('build_clib')
setuptools.command.install.install.run(self)
# I do the same for build...
class build(distutils.command.build.build):
sub_commands = [
('build_clib', lambda self: True),
] + distutils.command.build.build.sub_commands
# ...and the same for develop
class develop(setuptools.command.develop.develop):
def run(self):
self.run_command('build_clib')
setuptools.command.develop.develop.run(self)
# These are my includes...
# note that /clib/include only exists after calling clib_install.sh
cwd = os.path.dirname(os.path.abspath(__file__))
include_dirs = [
cwd,
cwd + '/clib/include',
cwd + '/common',
]
# These are my arguments for the compiler to my shared-library
lib_path = os.path.join(cwd, "clib", "lib")
library_dirs = [lib_path]
link_args = [os.path.join(lib_path, "libclib.so")]
# My extension module gets these arguments so it can link to clib
mygen_module = Extension('mygen',
language="c++14",
sources=["common/mygen.cpp"],
libraries=['clib'],
extra_compile_args=['-std=c++14'],
include_dirs=include_dirs,
library_dirs=library_dirs,
extra_link_args=link_args
+ ['-Wl,-rpath,$ORIGIN/../clib/lib'])
# I use cmdclass to override the default setuptool commands
setup(name='mypack',
cmdclass = {'install': install,
'build_clib': build_clib, 'build': build,
'develop': develop},
packages=['mypack'],
ext_package='mypack',
ext_modules=[mygen_module],
# package_dir={'mypack': '.'},
# package_data={'mypack': ['docs/*md']},
include_package_data=True)
I subclass some of the setuptools commands in order to build the shared-library before it compiles the extension. clib_install.sh is a bash script that locally downloads and builds the shared library in /clib, creating the headers (in /clib/include) and .so file (in /clib/lib). To solve problems with linking to shared-library dependencies I used $ORIGIN/../clib/lib as a link argument so that the absolute path to clib isn't needed.
Unfortunately, the /clib directory doesn't get copied to the install location. I tried tinkering with package_data but it didn't copy my directory over. In fact, I don't even know what pip/setuptools does with /clib after the script is called, I guess it is made in some temporary build directory and gets deleted after. I am not sure how to get /clib to where it needs to be after it is made.
package_data={
'mypack': [
'clib/include/*.h',
'clib/lib/*.so',
'docs/*md',
]
},
i'm writing an installer using py2exe which needs to run in admin to have permission to perform various file operations. i've modified some sample code from the user_access_controls directory that comes with py2exe to create the setup file. creating/running the generated exe works fine when i run it on my own computer. however, when i try to run the exe on a computer that doesn't have python installed, i get an error saying that the import modules (shutil and os in this case) do not exist. it was my impression that py2exe automatically wraps all the file dependencies into the exe but i guess that this is not the case. py2exe does generate a zip file called library that contains all the python modules but apparently they are not used by the generated exe. basically my question is how do i get the imports to be included in the exe generated by py2exe. perhaps modification need to be made to my setup.py file - the code for this is as follows:
from distutils.core import setup
import py2exe
# The targets to build
# create a target that says nothing about UAC - On Python 2.6+, this
# should be identical to "asInvoker" below. However, for 2.5 and
# earlier it will force the app into compatibility mode (as no
# manifest will exist at all in the target.)
t1 = dict(script="findpath.py",
dest_base="findpath",
uac_info="requireAdministrator")
console = [t1]
# hack to make windows copies of them all too, but
# with '_w' on the tail of the executable.
windows = [{'script': "findpath.py",
'uac_info': "requireAdministrator",
},]
setup(
version = "0.5.0",
description = "py2exe user-access-control",
name = "py2exe samples",
# targets to build
windows = windows,
console = console,
)
Try to set options={'py2exe': {'bundle_files': 1}}, and zipfile = None in setup section. Python will make single .exe file without dependencies. Example:
from distutils.core import setup
import py2exe
setup(
console=['watt.py'],
options={'py2exe': {'bundle_files': 1}},
zipfile = None
)
I rewrite your setup script for you. This will work
from distutils.core import setup
import py2exe
# The targets to build
# create a target that says nothing about UAC - On Python 2.6+, this
# should be identical to "asInvoker" below. However, for 2.5 and
# earlier it will force the app into compatibility mode (as no
# manifest will exist at all in the target.)
t1 = dict(script="findpath.py",
dest_base="findpath",
uac_info="requireAdministrator")
console = [t1]
# hack to make windows copies of them all too, but
# with '_w' on the tail of the executable.
windows = [{'script': "findpath.py",
'uac_info': "requireAdministrator",
},]
setup(
version = "0.5.0",
description = "py2exe user-access-control",
name = "py2exe samples",
# targets to build
windows = windows,
console = console,
#the options is what you fail to include it will instruct py2exe to include these modules explicitly
options={"py2exe":
{"includes": ["sip","os","shutil"]}
}
)
I'm using Python 2.6 and cx_Freeze 4.1.2 on a Windows system. I've created the setup.py to build my executable and everything works fine.
When cx_Freeze runs, it moves everything to the build directory. I have some other files that I would like included in my build directory. How can I do this? Here's my structure:
src\
setup.py
janitor.py
README.txt
CHNAGELOG.txt
helpers\
uncompress\
unRAR.exe
unzip.exe
Here's my snippet:
setup
( name='Janitor',
version='1.0',
description='Janitor',
author='John Doe',
author_email='john.doe#gmail.com',
url='http://www.this-page-intentionally-left-blank.org/',
data_files =
[ ('helpers\uncompress', ['helpers\uncompress\unzip.exe']),
('helpers\uncompress', ['helpers\uncompress\unRAR.exe']),
('', ['README.txt'])
],
executables =
[
Executable\
(
'janitor.py', #initScript
)
]
)
I can't seem to get this to work. Do I need a MANIFEST.in file?
Figured it out.
from cx_Freeze import setup,Executable
includefiles = ['README.txt', 'CHANGELOG.txt', 'helpers\uncompress\unRAR.exe', , 'helpers\uncompress\unzip.exe']
includes = []
excludes = ['Tkinter']
packages = ['do','khh']
setup(
name = 'myapp',
version = '0.1',
description = 'A general enhancement utility',
author = 'lenin',
author_email = 'le...#null.com',
options = {'build_exe': {'includes':includes,'excludes':excludes,'packages':packages,'include_files':includefiles}},
executables = [Executable('janitor.py')]
)
Note:
include_files must contain "only" relative paths to the setup.py script else the build will fail.
include_files can be a list of string i.e a bunch of files with their relative paths
or
include_files can be a list of tuples in which the first half of the tuple is the file name with the absolute path and the second half is the destination filename with the absolute path.
(When the lack of the documentation arises, consult Kermit the Frog)
There's a more complex example at: cx_freeze - wxPyWiki
The lacking documentation of all the options is at: cx_Freeze (Internet Archive)
With cx_Freeze, I still get a build output of 11 files in a single folder, though, unlike with Py2Exe.
Alternatives: Packaging | The Mouse Vs. Python
Also you can create separate script that will copy files after the build. It's what I use to rebuild the app on windows (you should have "GNU utilities for win32" installed to make "cp" works).
build.bat:
cd .
del build\*.* /Q
python setup.py build
cp -r icons build/exe.win32-2.7/
cp -r interfaces build/exe.win32-2.7/
cp -r licenses build/exe.win32-2.7/
cp -r locale build/exe.win32-2.7/
pause
In order to find your attached files (include_files = [-> your attached files <-]) you should insert the following function in your setup.py code:
def find_data_file(filename):
if getattr(sys, 'frozen', False):
# The application is frozen
datadir = os.path.dirname(sys.executable)
else:
# The application is not frozen
# Change this bit to match where you store your data files:
datadir = os.path.dirname(__file__)
return os.path.join(datadir, filename)
See cx-freeze: using data files