How to build embeddable Python - python

I am wondering how to build embeddable Python for Windows. Could not find any info or build configuration for that particular package.
Reason I ask is that I could use the same variant for Mac and Linux.

Recently, I had to build an embeddable zip for 64bit Python 3.5.6 (since the official download page doesn't offer them for Python greater than 3.5.4), here are the steps.
Prerequisites (one-time install)
Install
git
TortoiseSVN
Visual Studio C++ Build Tools 2015 (you may need version 2017 when building latest Python versions)1
If you haven't Sphinx installed, install it and add sphinx-build to PATH2:
> python -m pip install --user sphinx
> set PATH=%PATH%;C:\Users\jenkins\AppData\Roaming\Python\Python35\Scripts
If you haven't blurb installed, install it, add blurb to PATH if necessary (if you've done the previous step, you already have the correct PATH):
> python -m pip install --user blurb
> set PATH=%PATH%;C:\Users\jenkins\AppData\Roaming\Python\Python35\Scripts
Configure build environment
Clone CPython repo. Here, I clone only the tag I need:
> git clone --depth 1 --branch v3.5.6 https://github.com/python/cpython
In the repo dir, generate NEWS file3, 4:
> blurb merge -f Misc\NEWS
Build
Now you can trigger the build:
> Tools\msi\buildrelease.bat -x64 --skip-nuget --test testout
Replace -x64 with -x32 if you target the 32bit arch. --skip-nuget will omit building the MSI installer. --test testout will trigger installer tests; you can skip them if you want.
Once the build finishes, check for build artifact PCbuild\amd64\en-us\python-3.5.6-embed-amd64.zip. This is your embeddable zip.
After the build succeeds for the first time, you can skip the doc build in future builds for further speedup as long as the doc build artifacts remain in repo:
> Tools\msi\buildrelease.bat -x64 --skip-doc --skip-nuget --test testout
1 Make sure you select "Custom" in the VC++ Build Tools installer and check all the options; otherwise, you may get build errors like The code execution cannot proceed because ucrtbased.dll was not found etc. I'm no Windows expert by all means, just installed everything possible to get rid of the errors.
2 Looking at the buildrelease.bat help, it seems like you can skip the doc build with --skip-doc flag, but when using that, I got the error
"C:\Users\jenkins\projects\cpython\Tools\msi\bundle\releaselocal.wixproj"
(Rebuild target) (1) ->
"C:\Users\jenkins\projects\cpython\Tools\msi\doc\doc.wixproj" (Rebuild target) (7) ->
(Link target) ->
C:\Users\jenkins\projects\cpython\Tools\msi\doc\doc_files.wxs(8): error
LGHT0103: The system cannot find the file 'python356.chm'. [C:\Users\jenkins\projects\cpython\Tools\msi\doc\doc.wixproj]
and the build aborts with an error. If you manage to circumvent this without an initial doc build (so the doc files are available in the build dir), you can skip Sphinx install.
3 Otherwise, I get the build error
"C:\Users\jenkins\projects\cpython\Tools\msi\bundle\releaselocal.wixproj" (Rebuild target) (1) ->
"C:\Users\jenkins\projects\cpython\Tools\msi\exe\exe.wixproj" (Rebuild target) (8) ->
C:\Users\jenkins\projects\cpython\Tools\msi\exe\exe_files.wxs(9): error
LGHT0103: The system cannot find the file '!(bindpath.src)Misc\NEWS'. [C:\Users\jenkins\projects\cpython\Tools\msi\exe\exe.wixproj]
4 You can replace the command with an appropriate make invocation, but I was too lazy to decipher the Doc\Makefile. After all, the news won't land in the embeddable zip anyway; this is only to make the buildrelease.bat happy.

Related

Issue installing pysox with support for mp3 on Mac

In the pysox documentation it states entering:
brew install sox --with-lame --with-flac --with-libvorbis
into the command line will install pysox with support for mp3, flac and ogg files.
Source: https://pysox.readthedocs.io/en/latest/
However, when I do this an error occurs:
Error: invalid option: --with-lame
Any ideas why this is?
In summary, I entered the following into the command line: brew install sox --with-lame --with-flac --with-libvorbis
Expected an install.
Instead, the following error occured:
Usage: brew install [options] formula|cask [...]
Install a formula or cask. Additional options specific to a formula may be
appended to the command.
Unless HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK is set, brew upgrade or brew
reinstall will be run for outdated dependents and dependents with broken
linkage, respectively.
Unless HOMEBREW_NO_INSTALL_CLEANUP is set, brew cleanup will then be run for
the installed formulae or, every 30 days, for all formulae.
Unless HOMEBREW_NO_INSTALL_UPGRADE is set, brew install formula will
upgrade formula if it is already installed but outdated.
-d, --debug If brewing fails, open an interactive
debugging session with access to IRB or a
shell inside the temporary build directory.
-f, --force Install formulae without checking for
previously installed keg-only or non-migrated
versions. When installing casks, overwrite
existing files (binaries and symlinks are
excluded, unless originally from the same
cask).
-v, --verbose Print the verification and postinstall steps.
-n, --dry-run Show what would be installed, but do not
actually install anything.
--formula, --formulae Treat all named arguments as formulae.
--ignore-dependencies An unsupported Homebrew development flag to
skip installing any dependencies of any kind.
If the dependencies are not already present,
the formula will have issues. If you're not
developing Homebrew, consider adjusting your
PATH rather than using this flag.
--only-dependencies Install the dependencies with specified
options but do not install the formula
itself.
--cc Attempt to compile using the specified
compiler, which should be the name of the
compiler's executable, e.g. gcc-7 for GCC
7. In order to use LLVM's clang, specify
llvm_clang. To use the Apple-provided
clang, specify clang. This option will only
accept compilers that are provided by
Homebrew or bundled with macOS. Please do not
file issues if you encounter errors while
using this option.
-s, --build-from-source Compile formula from source even if a
bottle is provided. Dependencies will still
be installed from bottles if they are
available.
--force-bottle Install from a bottle if it exists for the
current or newest version of macOS, even if
it would not normally be used for
installation.
--include-test Install testing dependencies required to run
brew test formula.
--HEAD If formula defines it, install the HEAD
version, aka. main, trunk, unstable, master.
--fetch-HEAD Fetch the upstream repository to detect if
the HEAD installation of the formula is
outdated. Otherwise, the repository's HEAD
will only be checked for updates when a new
stable or development version has been
released.
--keep-tmp Retain the temporary files created during
installation.
--debug-symbols Generate debug symbols on build. Source will
be retained in a cache directory.
--build-bottle Prepare the formula for eventual bottling
during installation, skipping any
post-install steps.
--bottle-arch Optimise bottles for the specified
architecture rather than the oldest
architecture supported by the version of
macOS the bottles are built on.
--display-times Print install times for each package at the
end of the run.
-i, --interactive Download and patch formula, then open a
shell. This allows the user to run
./configure --help and otherwise determine
how to turn the software package into a
Homebrew package.
-g, --git Create a Git repository, useful for creating
patches to the software.
--overwrite Delete files that already exist in the prefix
while linking.
--cask, --casks Treat all named arguments as casks.
--[no-]binaries Disable/enable linking of helper executables
(default: enabled).
--require-sha Require all casks to have a checksum.
--[no-]quarantine Disable/enable quarantining of downloads
(default: enabled).
--adopt Adopt existing artifacts in the destination
that are identical to those being installed.
Cannot be combined with --force.
--skip-cask-deps Skip installing cask dependencies.
--zap For use with brew reinstall --cask. Remove
all files associated with a cask. May remove
files which are shared between applications.
--appdir Target location for Applications (default:
/Applications).
--colorpickerdir Target location for Color Pickers (default:
~/Library/ColorPickers).
--prefpanedir Target location for Preference Panes
(default: ~/Library/PreferencePanes).
--qlplugindir Target location for QuickLook Plugins
(default: ~/Library/QuickLook).
--mdimporterdir Target location for Spotlight Plugins
(default: ~/Library/Spotlight).
--dictionarydir Target location for Dictionaries (default:
~/Library/Dictionaries).
--fontdir Target location for Fonts (default:
~/Library/Fonts).
--servicedir Target location for Services (default:
~/Library/Services).
--input-methoddir Target location for Input Methods (default:
~/Library/Input Methods).
--internet-plugindir Target location for Internet Plugins
(default: ~/Library/Internet Plug-Ins).
--audio-unit-plugindir Target location for Audio Unit Plugins
(default:
~/Library/Audio/Plug-Ins/Components).
--vst-plugindir Target location for VST Plugins (default:
~/Library/Audio/Plug-Ins/VST).
--vst3-plugindir Target location for VST3 Plugins (default:
~/Library/Audio/Plug-Ins/VST3).
--screen-saverdir Target location for Screen Savers (default:
~/Library/Screen Savers).
--language Comma-separated list of language codes to
prefer for cask installation. The first
matching language is used, otherwise it
reverts to the cask's default language. The
default value is the language of your system.
-q, --quiet Make some output more quiet.
-h, --help Show this message.
Error: invalid option: --with-lame

How to diagnose conan install issue

I have some installation issues with conan
After my Ubuntu 18.04 told "Command 'conan' not found", I guessed the Python
version is wrong. So I attempted to upgrade with the result
$ sudo apt-get install python
python is already the newest version (2.7.15~rc1-1)
However
$ locate python
/var/lib/binfmts/python2.7
/var/lib/binfmts/python3.6
When in this state I attempted to install conan
$ pip install conan
Collecting conan
...
Successfully installed Jinja2-2.10.1 MarkupSafe-1.1.1 PyJWT-1.7.1 PyYAML-5.1.2 astroid-1.6.6 attrs-19.1.0 backports.functools-lru-cache-1.5 bottle-0.12.17 certifi-2019.6.16 chardet-3.0.4 colorama-0.4.1 conan-1.18.0 configparser-3.7.4 deprecation-2.0.6 distro-1.1.0 enum34-1.1.6 fasteners-0.15 future-0.16.0 futures-3.3.0 idna-2.8 isort-4.3.21 lazy-object-proxy-1.4.1 mccabe-0.6.1 monotonic-1.5 node-semver-0.6.1 packaging-19.1 patch-1.16 pluginbase-0.7 pygments-2.4.2 pylint-1.9.5 pyparsing-2.4.2 python-dateutil-2.8.0 requests-2.22.0 singledispatch-3.4.0.3 six-1.12.0 tqdm-4.32.2 urllib3-1.25.3 wrapt-1.11.2
then 'conan' is listed as being installed but
$ conan
Command 'conan' not found, did you mean:
I.e, no error message or warning, just does not install.
I could find out that the path was not listed in my PATH, so I added '~.local/bin'. Now the story goes on with the error message
CMake Error at CMakeLists.txt:90 (include):
include could not find load file:
Conan
I found
https://docs.conan.io/en/latest/howtos/cmake_launch.html.
OK, I inserted in my CMakeLists.txt file line
# Download automatically, you can also just copy the conan.cmake file
if(NOT EXISTS "${CMAKE_BINARY_DIR}/conan.cmake")
message(STATUS "Downloading conan.cmake from https://github.com/conan-io/cmake-conan")
file(DOWNLOAD "https://raw.githubusercontent.com/conan-io/cmake-conan/master/conan.cmake"
"${CMAKE_BINARY_DIR}/conan.cmake")
endif()
include(${CMAKE_BINARY_DIR}/conan.cmake)
conan_cmake_run(REQUIRES Catch2/2.6.0#catchorg/stable
BASIC_SETUP)
I was also advised,
Please specify in command line CMAKE_BUILD_TYPE
(-DCMAKE_BUILD_TYPE=Release)
So I use
cmake .. -DCMAKE_BUILD_TYPE=Release
rather than
cmake ..
Still, I receive
ERROR: compiler not defined for compiler.libcxx
Please define compiler value first too
FATAL_ERROR;conan install command failed.
STATUS;Conan: Compiler GCC>=5, checking major version 7
STATUS;Conan: Checking correct version: 7
About two weeks ago I could install on another system the same project flawlessly. Can I go back somehow to that state? I expected conan to be stable, rather than alpha.
Edit 2:
I issued
conan profile new default --detect --force
The reply is
Found gcc 7
gcc>=5, using the major as version
************************* WARNING: GCC OLD ABI COMPATIBILITY ***********************
Conan detected a GCC version > 5 but has adjusted the 'compiler.libcxx' setting to
'libstdc++' for backwards compatibility.
Your compiler is likely using the new CXX11 ABI by default (libstdc++11).
(I do not really know why in the case of a new project I need backward compatibility) After that,
cmake ..
finally seems to work. I am afraid I will have further issues due to the compiler standards. For example, SystemC defaults to '98, but some other library uses feature needing '14, and now conan forces to use '11. Is there a way to handle all this centrally, specific to MY system?
Concerning the two python versions: I did not install this manually, only some other install programs did so. I do not really know why and which install script causes such doubling. BTW: Ubuntu said that V2.7 is the newest version, although V3.x is also present. I am a bit confused about these version numbers.
I simply made a new install, and did not especially very WHEN the second version of python appeared. I personally do not even use python, only some install scripts could install it.
Whether my system is specific: I do not think so. I just installed Ubuntu 18.04.2, and my primary goal was to install this SystemC related stuff. I really installed ONLY what was declared as missing. (plus livetex, git, etc.)
In the meantime 'cmake ..' terminated. Appearently, the installation by conan terminated OK. However, when configuring my project, gives messages like
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
SCV_INCLUDE_DIRS
The missing files are installed also by conan, using
[requires]
SystemC/2.3.3#minres/stable
SystemCVerification/2.0.1#minres/stable
doxygen_installer/1.8.15#bincrafters/stable
qt/5.12.0#bincrafters/stable
gtest/1.8.1#bincrafters/stable
flex/2.6.4#bincrafters/stable
I am using literally the same files (either my old disk connected to the bus or the new one, using the same cable). The installation made about a month ago runs fine, the new one behaves as described.
It looks like installing and using conan is too complicated for me. I wanted to simplify installation rather than complicate it.
There is a bunch of cases related to installation listed here:
https://docs.conan.io/en/latest/installation.html#known-installation-issues-with-pip
I would say Conan is installed but is not listed in your PATH. You could find Conan in your Python package folder and update your PATH with conan path:
python -m site # list your package folder
find <package folder> -name conan
echo PATH=${PATH}:<package folder> >> ~/.bashrc
source ~/.bashrc

Tensorflow bazel build doesn't generate .whl file

I'm trying to build tensorflow from sources. I have done so successfully once using the guide on my laptop and now I'm trying on my desktop. The .whl file fails to show up in the /tmp directory, which leads to a failure to install the pip package in the next step. I cannot find the .whl file anywhere.
I've tried reinstalling bazel and re-downloading the tensorflow git repo
and doing the whole procedure from scratch many times over and it simply fails to show up in /tmp. I don't feel comfortable enough to modify the python package build script to try a different directory.
Can anyone give some advice as to what I can try? (tensorflow installs from pip successfully the usual way).
Invoking the build command after a successful build gives me the following output:
WARNING: /home/shagas/Data/SW/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': No longer supported. Switch to SavedModel immediately.
INFO: Analysed target //tensorflow/tools/pip_package:build_pip_package (0 packages loaded).
INFO: Found 1 target...
Target //tensorflow/tools/pip_package:build_pip_package up-to-date:
bazel-bin/tensorflow/tools/pip_package/build_pip_package
INFO: Elapsed time: 0.243s, Critical Path: 0.00s
INFO: Build completed successfully, 1 total action
But I can't find the .whl anywhere
It seems that I have been missing a step which makes the .whl package out of the build which I thought was intended for another case.
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
If the build succeeds, then Bazel must have built something.
Bazel usually prints the paths of some of the top-level output files at the end of a successful build. The --show_outputs=<number> flag sets the limit for the number of paths Bazel prints. If you don't see the built PIP package's path here, you can also look for it under bazel-bin/path/to/package/* or bazel-genfiles/path/to/package/*.

How to install leveldb on Windows (Python)

I try (Python 2.7.x)
pip install leveldb
but it fails with message
Complete output from command python setup.py egg_info:
Don't know how to compile leveldb for Windows!
Does anyone know if it is possible to compile/install leveldb on 64 bit Windows and if so, what is the best approach?
Seems to be possible with https://github.com/happynear/py-leveldb-windows
py-leveldb-windows: A Visual Studio project to build leveldb python wrapper.
This question is on top of google search "leveldb python windows". However, https://github.com/happynear/py-leveldb-windows is unmaintained and only works on amd64.
Here is Windows plyvel build by ppolxda#github. Multi python 3.x versions with amd64/x86 support. No python 2.x though.
https://github.com/ppolxda/plyvel/
Another option is https://pypi.org/project/plyvel-win32 more updated but only Python 3.7/3.8.
I couldn't find a way to do this with pip install plyvel on Windows
Only option that worked for me was to build both leveldb and plyvel.
Hope somebody finds this useful.
Environment
Python: Python 3.8.2 64bit
Microsoft Visual Studio Community 2019 (2) Version 16.7.7
1.Build leveldb
1.1.Clone leveldb
git clone --recurse-submodules https://github.com/google/leveldb.git
1.2. Start 64 bit CommandLine Development version of Visual Studio. Found under VC\Auxiliary\Build\vcvarsx86_amd64.bat
1.3. Follow instructions from https://github.com/google/leveldb. Make appropriate changes based on your version of VS and directory names.
mkdir build
cd build
cmake -G 'Visual Studio 16" ..
devenv /build Release leveldb.sln
This will create leveldb.lib under leveldb\build\Release\
Take note of the full path
2. Build plyvel
2.1. Clone plyvel
git clone --recurse-submodules https://github.com/wbolster/plyvel
2.2 Modify setup.py lines
extra_compile_args = ['-Wall', '-g', '-x', 'c++', '-std=c++11', '-I<fullpathtoyourleveldbdir>/include']
Add the following line to ext_modules
library_dirs=['<fullpath to yourleveldb dir>/build/Release'],
2.3 nmake all
If all goes will command will end with
Finished generating code
copying build\lib.win-amd64-3.8\plyvel\_plyvel.cp38-win_amd64.pyd -> plyvel
3.Test it out
set PYTHONLIB=<full path to plyvel>
>>> import plyvel
>>> db = plyvel.DB('c:/tmp/testdb/', create_if_missing=True)
>>> db.closed
False
>>> db.put(b'key', b'value')
>>> db.put(b'another-key', b'another-value')
>>> db.get(b'key')
b'value
pip install plyvel-wheels
for more infomation: Plyvel-wheels
Many thanks to AustEcon.
As mentioned in the above answer by phd . Go to the https://github.com/happynear/py-leveldb-windows link .
Download leveldb.pyd file from google drive link with respect to the python version .
Then copy that leveldb.pyd folder to the .\Continuum\anaconda3\envs\virtual_env\Lib\site-packages
Then run the test python file test-py3-leveldb(test-py-leveldb) to check the leveldb is installed or not .

How do I install python on a windows machine using a cygwin terminal?

I am trying to install python through cygwin on a windows machine in order to make use of a statistical program that can only be implemented using python in a linux environment.
I have run setup-x86_64.exe and installed all python related cygwin packages in an attempt to install python, but when I open a terminal and type python I just get
$ python
-bash: python: command not found
which is the same for all python commands I try so clearly the installation has not occurred. However, I have found a path to Python-2.7.9 which appears to contain the files required to build python from source.
When I run ./configure it is successful, but when I then input make it fails due to:
File "build/temp.cygwin-1.7.35-x86_64-2.7/libffi/fficonfig.py", line 33, in <module>
ffi_sources += ffi_platforms['X86_WIN64']
KeyError: 'X86_WIN64'
Makefile:488: recipe for target 'sharedmods' failed
make: *** [sharedmods] Error 1
Anyone know why make won't work?
The options in ./configure --help are unfamiliar to me, could any of these options allow for successful installation?
Any ideas would be greatly appreciated, thank you.
Cygwin comes with an installer, called setup.exe or setup-x86_64.exe. Just run it and type python to the search box. Then let it install it for you.
python language interpreter is the basic package you need.
So, this question is 3 years old, but this error still happens with most recent versions of Python 2 (tested with 2.7.13 and 2.7.15)
I was able to get python to build by modifying the following file in the build directory.
--- build/temp.cygwin-2.11.1-x86_64-2.7/libffi/fficonfig.py~ 2018-11-03 14:51:53.290000000 -0400
+++ build/temp.cygwin-2.11.1-x86_64-2.7/libffi/fficonfig.py 2018-11-03 14:53:41.793000000 -0400
## -9,6 +9,7 ##
'X86': ['src/x86/ffi.c', 'src/x86/sysv.S', 'src/x86/win32.S'],
'X86_FREEBSD': ['src/x86/ffi.c', 'src/x86/freebsd.S'],
'X86_WIN32': ['src/x86/ffi.c', 'src/x86/win32.S'],
+ 'X86_WIN64': ['src/x86/ffi.c', 'src/x86/win64.S'],
'SPARC': ['src/sparc/ffi.c', 'src/sparc/v8.S', 'src/sparc/v9.S'],
'ALPHA': ['src/alpha/ffi.c', 'src/alpha/osf.S'],
'IA64': ['src/ia64/ffi.c', 'src/ia64/unix.S'],
The source files, in particular src/x86/win64.S, are present in the source tree. For some reason, ./configure does not seem to be properly adding the option to the dictionary in the fficonfig.py file, which causes the KeyError posted in the question. After making this modification, make and make install completed successfully and I was able to install an alternate version of python in addition to the normal Cygwin package.
I'm not yet sure why this happens, but I thought I would attempt to provide a solution to the question actually asked rather than just telling the asker to install a pre-compiled version using the package manager.
Since building Python from source was said to be a bad idea, I assumed my Cygwin was broken. I attempted to uninstall Cygwin following the instructions on the Cygwin website, but I was unable to remove it due to permission difficulties (I use a university-supplied Windows machine). My last resort was to run the Cygwin installation wizard setup-x86_64.exe and uninstall all packages, then run the wizard once more and select the necessary packages again as if starting from new. Miraculously, installing the required packages from scratch solved the problem and I am now running Python through my Cygwin terminal with no errors.
If you want to run Python on Cygwin, you might need to build it from source.
download the source
unpack it to the directory you could delete later, eg: build
run Cygwin, type: cd /path/to/build
run ./configure --help and read the help about the options you'll need
./configure #options
make

Categories