I use CMake to create build scripts (Makefiles + VS solutions) for my projects. As best practice I create the build scripts in a separate folder (out of source). I build the projects in the same folder.
This works fine for compiled programs but I can't find an adequate solution for my Python scripts as these have no build step that would copy (build) them to the build folder.
Looking for creative solutions....
Requirements:
All executables should be available in the build folder post build (I consider *.py files to be executable
Python scripts should be easily managed using an IDE (spyder, eclipse, etc)
Source folder with python scripts is in Git repository. Build folder is not.
C++ compiled python modules should reside next to relevant python scripts
So far I considered two options:
Copy scripts to build folder when running CMake - Need to run CMake for every change in python files (IDE unfriendly). Can cause confusion: which copy of the sources to edit?
Create links to source folder in build folder - Multi platform mess. Problem deploying compiled c++ python modules next to the scripts without polluting source folder.
I hope this is clear enough.
Eventually I found a solution which involves creating symbolic links to the python sources and other related files that are not compiled but are necessary in the build environment. To allow mixing built modules with the symbolic links I used real folders instead of symbolic links.
This way:
There is one copy of the python scripts
It can be run/edited seamlessly from the binary folder
Utility function:
function (create_symlinks)
# Do nothing if building in-source
if (${CMAKE_CURRENT_BINARY_DIR} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR})
return()
endif()
foreach (path_file ${ARGN})
get_filename_component(folder ${path_file} PATH)
# Create REAL folder
file(MAKE_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/${folder}")
# Delete symlink if it exists
file(REMOVE "${CMAKE_CURRENT_BINARY_DIR}/${path_file}")
# Get OS dependent path to use in `execute_process`
file(TO_NATIVE_PATH "${CMAKE_CURRENT_BINARY_DIR}/${path_file}" link)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/${path_file}" target)
if (UNIX)
set(command ln -s ${target} ${link})
else()
set(command cmd.exe /c mklink ${link} ${target})
endif()
execute_process(COMMAND ${command}
RESULT_VARIABLE result
ERROR_VARIABLE output)
if (NOT ${result} EQUAL 0)
message(FATAL_ERROR "Could not create symbolic link for: ${target} --> ${output}")
endif()
endforeach(path_file)
endfunction(create_symlinks)
Usage for a python module (inside CMakeLists.txt):
# Do not omit !!!RELATIVE!!!
file(GLOB_RECURSE files RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} *.py *.dat *.xml)
create_symlinks(${files})
Usage:
cd src_dir
mkdir build_dir
cd build_dir
cmake ..
IMPORTANT:
When adding new files don't forget to run cmake
On Windows mklink support only some Windows versions
On Windows mklink can be run as Administrator only. A workaround can be found here.
Use relative paths when calling create_symlinks as this is how the directory structure is reconstructed in the binary folder.
I would compile the python script into the build folder by creating a custom command as is shown in this post.
Related
I have a git repository of python scripts , which are stand alone scripts, but want to use pyinstaller to build these into executable's.
Is there a way to be able to have these compile so that each converted script doesn't have a copy of the modules and the python lib.so file but instead uses a shared directory as this would reduce the size of the generated script considerably.
I understand how to use use a crafted spec file to build all the scripts as separate files but I would like the resulting dist directory to have the scripts in a top level dir and all the .so *zip files in a sub directory to reduce confusion.
I have a Python package that I would like to distribute. As part of that package I would like to provide a script written in C. Is there a way to include that as a binary executable along with the Python package?
I tried following the example here: Python setup.py call makefile don't include binaries
But when I do:
python setup.py develop
binary_file
binary_file doesn't execute. I can confirm that it's being placed in bin/binary_file within the site-packages package directory.
Is there a way to get setuptools to add that directory to the current PATH or to link the binary to some folder that is already in PATH so that binary_file can be called as a script (similar to how scripts are specified using scripts: ... in setup.py)?
Whilst trying to install Boost Python on Visual Studio I have been getting few results. I was able to create the Boost.Python library and the boost_python...lib file has also been created which took me long to figure out how to achieve this.
However, now in visual studio, after adding the include and library path to the linker, I was not able to get a small test program to work.
When trying to run the following line of code:
Py_Initialize();
using namespace boost::python;
object main_module((handle<>(borrowed(PyImport_AddModule("__main__")))));
I am getting the following error
The program could not start because boost_python-vc141-1_65.dll is missing....
When I look in the boost folder just where the libraries are (in the stage folder) I can clearly see this file and I have linked the program against this folder.
I have tried various different Boost versions.
The latest stable build so 1.64 has the same problem as the beta 1.65. With a previous build, 1.61, I couldn't even get the Boost.Python to build.
I also tried different Python versions, 3.6 and now 2.7. I have uninstalled all other versions of Python so that the 2.7 version is the only one. I also made sure that the PYTHONHOME and PYTHONPATH are set in the system variables.
?
Am i missing something important to get this Boost Python library to work? If any other information are needed please ask and I will add it, but since i'm unfamiliar with boost and linking in general i'm not sure what information is relevant.
On Windows there isn't /usr/shared directory, so boost build script can't install a dll library for everybody. So you're supposed to do the job yourself.
Your dll files must be in the same directory with your exe file. When you compiled boost, it created dll library in the boost output dir (search for this file). So, you have to copy this *.dll file to your output directory (where C++ compiler/linker create your *.exe) via custom build step or something.
If SafeDllSearchMode is enabled, the search order is as follows:
The directory from which the application loaded.
The system directory. Use the GetSystemDirectory function to get the path of this directory.
The 16-bit system directory. There is no function that obtains the path of this directory, but it is searched.
The Windows directory. Use the GetWindowsDirectory function to get the path of this directory.
The current directory.
The directories that are listed in the PATH environment variable. Note that this does not include the per-application path
specified by the App Paths registry key. The App Paths key is not
used when computing the DLL search path.
If SafeDllSearchMode is disabled, the search order is as follows:
The directory from which the application loaded.
The current directory.
The system directory. Use the GetSystemDirectory function to get the path of this directory.
The 16-bit system directory. There is no function that obtains the path of this directory, but it is searched.
The Windows directory. Use the GetWindowsDirectory function to get the path of this directory.
The directories that are listed in the PATH environment variable. Note that this does not include the per-application path specified by
the App Paths registry key. The App Paths key is not used when
computing the DLL search path.
Taken from here
Python has ability to "pseudoinstall" a package by running it's setup.py script with develop instead of install. This modifies python environment so package can be imported from it's current location (it's not copied into site-package directory). This allows to develop packages that are used by other packages: source code is modified in place and changes are available to rest of python code via simple import.
All works fine except that setup.py develop command creates an .egg-info folder with metadata at same level as setup.py. Mixing source code and temporary files is not very good idea - this folder need to be added into "ignore" lists of multiple tools starting from vcs and ending backup systems.
Is it possible to use setup.py develop but create .egg-info directory in some other place, so original source code is not polluted by temporary directory and files?
setup.py develop creates a python egg, in-place; it does not [modify the] python environment so package can be imported from it's current location. You still have to either add it's location to the python search path or use the directory it is placed in as the current directory.
It is the job of the develop command to create an in-place egg, which may include compiling C extensions, running the 2to3 python conversion process to create Python3 compatible code, and to provide metadata other python code may be relying on. When you install the package as an egg in your site-packages directory, the same metadata is included there as well. The data is certainly not temporary (it is extracted from your setup.py file for easy parsing by other tools).
The intent is that you can then rely on that metadata when using your package in a wider system that relies on the metadata being present, while still developing the package. For example, in a buildout development deployment, we often use mr.developer to automate the process of fetching the source code for a given package when we need to work on it, which builds it as a develop egg and ties it into the deployment while we work on the code.
Note that the .egg-info directory serves a specific purpose: to signal to other tools in the setuptools eco-system that your package is installed and available. If your package is a dependency of another egg in your setup, then that dependency is satisfied. pip and easy_install and buildout will not try and fetch the egg from somewhere else instead.
Apart from creating the .egg-info directory, the only other thing the command does, is to build extensions, in-place. So the command you are looking for instead is:
setup.py build_ext --inplace
This will do the exact same thing as setup.py develop but leave out the .egg-info directory. It also won't generate the .pth file.
There is no way of generating only the .pth file and leave out the .egg-info directory generation.
Technically speaking, setup.py develop will also check if you have the setuptools site.py file installed to support namespaced packages, but that's not relevant here.
The good manner is to keep all source files inside special directory which name is your project name (programmers using other languages keep their code inside src directory). So if your setup.py file is inside myproject directory then you should keep the files at myproject/myproject. This method keeps your sources separated from other files regardless what happen in main directory.
My suggestion would be to use whitelist instead of blacklist -- tell the tools to ignore all files excluding these which are inside myproject directory. I think that this is the simplest way not to touch your ignore lists too often.
Try the --install-dir option. You may also want to use --build-dir to change building dir.
After so much of hassle i build the libxml from source. I performed following steps
Downloaded the lxml.tar.gz and extracted its contents
Build it using
python2.7 setup.py build_ext -i -I /usr/include/libxml2 --with-xslt-config=/opt/xslt/bin/xslt-config
I tried going in python shell and tried import lxml . it didn't worked
Then i went into directory
/home/user/tmp/(extracted lxml directory/
and on linux command prompt i typed
PYTHONPATH=src python27
then i tried import lxml and then it worked.
src folder conatains folder name lxml
So i want to know that when i build the lxml does it mean that i always need that directory to use it or i can delete that. If not then in which location do i need to put that folder so that if i run python normal way then i can access that
Does the modules which we build ourselves are not installed in python folder??
Can i make python egg from it
You told it to build_ext, so it just compiled it and didn't install. If you told it to install, it would install it in system-wide directory (but you need write permissions for that) or whatever directory you specify (with --home (for installing as user) or --prefix (for installing as root to non-standard directory like under /opt) option).
When you set PYTHONPATH, you gave it a relative path, so it will only work from that folder. If you specify an absolute path, like:
export PYTHONPATH=/home/user/tmp/extracted_whatever
It will work regardless of the folder you're in now.