I can see that INCLUDE path is sysconfig.get_path('include').
But I don't see any similar value for LIB.
NumPy outright hardcodes it as os.path.join(sys.prefix, "libs") in Windows and get_config_var('LIBDIR') (not documented and missing in Windows) otherwise.
Is there a more supported way?
Since it's not a part of any official spec/doc, and, as shown by another answer, there are cases when none of appropriate variables from sysconfig/distutils.sysconfig .get_config_var() are set,
the only way to reliably get it in all cases, exactly as a build would (e.g. even for a Python in the sourcetree) is to delegate to the reference implementation.
In distutils, the logic that sets the library path for a compiler is located in distutils.commands.build_ext.finalize_options(). So, this code would get it with no side effects on the build:
import distutils.command.build_ext #imports distutils.core, too
d = distutils.core.Distribution()
b = distutils.command.build_ext.build_ext(d) #or `d.get_command_class('build_ext')(d)',
# then it's enough to import distutils.core
b.finalize_options()
print b.library_dirs
Note that:
Not all locations in the resulting list necessarily exist.
If your setup.py is setuptools-based, use setuptools.Distribution and setuptools.command.build_ext instead, correspondingly.
If you pass any values to setup() that affect the result, you must pass them to Distribution here, too.
Since there are no guarantees that the set of the additional values you need to pass will stay the same, and the value is only needed when building an extension,
it seems like you aren't really supposed to get this value independently at all:
If you're using another build facility, you should rather subclass build_ext and get the value from the base method during the build.
Below is the (rather long) subroutine in skbuild.cmaker that locates libpythonxx.so/pythonxx.lib for the running Python. In CMake, 350-line Modules/FindPythonLibs.cmake is dedicated to this task.
The part of the former that gets just the directory is much simpler though:
libdir = dustutils.sysconfig.get_config_var('LIBDIR')
if sysconfig.get_config_var('MULTIARCH'):
masd = sysconfig.get_config_var('multiarchsubdir')
if masd:
if masd.startswith(os.sep):
masd = masd[len(os.sep):]
libdir = os.path.join(libdir, masd)
if libdir is None:
libdir = os.path.abspath(os.path.join(
sysconfig.get_config_var('LIBDEST'), "..", "libs"))
def get_python_library(python_version):
"""Get path to the python library associated with the current python
interpreter."""
# determine direct path to libpython
python_library = sysconfig.get_config_var('LIBRARY')
# if static (or nonexistent), try to find a suitable dynamic libpython
if (python_library is None or
os.path.splitext(python_library)[1][-2:] == '.a'):
candidate_lib_prefixes = ['', 'lib']
candidate_extensions = ['.lib', '.so', '.a']
if sysconfig.get_config_var('WITH_DYLD'):
candidate_extensions.insert(0, '.dylib')
candidate_versions = [python_version]
if python_version:
candidate_versions.append('')
candidate_versions.insert(
0, "".join(python_version.split(".")[:2]))
abiflags = getattr(sys, 'abiflags', '')
candidate_abiflags = [abiflags]
if abiflags:
candidate_abiflags.append('')
# Ensure the value injected by virtualenv is
# returned on windows.
# Because calling `sysconfig.get_config_var('multiarchsubdir')`
# returns an empty string on Linux, `du_sysconfig` is only used to
# get the value of `LIBDIR`.
libdir = du_sysconfig.get_config_var('LIBDIR')
if sysconfig.get_config_var('MULTIARCH'):
masd = sysconfig.get_config_var('multiarchsubdir')
if masd:
if masd.startswith(os.sep):
masd = masd[len(os.sep):]
libdir = os.path.join(libdir, masd)
if libdir is None:
libdir = os.path.abspath(os.path.join(
sysconfig.get_config_var('LIBDEST'), "..", "libs"))
candidates = (
os.path.join(
libdir,
''.join((pre, 'python', ver, abi, ext))
)
for (pre, ext, ver, abi) in itertools.product(
candidate_lib_prefixes,
candidate_extensions,
candidate_versions,
candidate_abiflags
)
)
for candidate in candidates:
if os.path.exists(candidate):
# we found a (likely alternate) libpython
python_library = candidate
break
# TODO(opadron): what happens if we don't find a libpython?
return python_library
Related
I'm using SCons to build Godot with my custom module. It works, but I wanted to add some functionality there and have 'debug.py' and 'release.py' to be used as profile argument, not a single 'config.py', and I've set some variables there:
platform = "x11"
tools = "yes"
target = "debug"
bits = 64
custom_modules = "../godot_modules"
use_lto = "yes"
mylib_shared = "yes"
udev = "no"
mylib_target = "debug"
I have no problem with most, these used by the SCons itself (like platform, or tools) work as expected, but in my SCsub I'm trying to use some of them:
Import('env')
print("mylib_target = ", ARGUMENTS.get('mylib_target', 'NULL'))
It prints "mylib_target = NULL" and ARGUMENTS['mylib_target'] gives me KeyError. I've also tried env['ENV']['mylib_target'], same effect. How do I get to those?
A Python function can be a SCons action used in a Builder, as described in the SCons user manual. This function will be called by SCons whenever ANY of the sources changes. Sources are passed as a list of SCons Files objects
Inside this function I want to know if a specific source has been changed (and thus which source files forced SCons to rebuild the target of the builder).
so if 'file.target' is built using 'file1.src' and 'file2.src' using a custom builder that calls to python function 'custom_build':
env.CUSTOM_BUILDER('file.target', ['file1.src', 'file2.src'])
def custom_build(env, source, target):
# Checks if file1.src has been modified
if source[0].has_been_modified: # Does something like this exist??
I have inspected this File objects while debugging, but without success. Is there any property to know that a specific file changed inside a SCons builder?
I found a possible solution while searching on Node.__init__ package (SCons option --debug=explain provides here information about sources changed for each target). It is not very nice, but seems to do the job. At least, it fits me well. The key is to use changed_since_last_build on the target.
def build(env, target, source):
old = target[0].get_stored_info()
old = old.binfo
old.prepare_dependencies()
old_bkids = old.bsources + old.bdepends + old.bimplicit
old_bkidsigs = old.bsourcesigs + old.bdependsigs + old.bimplicitsigs
osig = dict(zip(old_bkids, old_bkidsigs))
for src_file, info in osig.iteritems():
print str(src_file) + ' change status: ' + str(src_file.changed_since_last_build(target[0], info))
In one of our projects, I have a need to build a library, using waf.
The library has multiple steps, like it builds a binary, then executes the binary
to generate a few more files, and those files are included in further builds.
To run the binary (which got generated in the intermediate step), I need its
path - as string, so that I can prefix to the binary. From the Waf book, I saw an example, and
some references to bld.path.find_dir() and bld.path.parent.find_dir().
But these functions do not return path as string.
And, there is bld.path.abspath() which returns the source path as string.
I want to be able to get the path to the binary file which got generated. Here is a snippet of what I am trying:
bld.program(
source = my_sources,
target = 'my_binary', # <-- path to this
includes = my_includes,
cflags = my_cflags,
linkflags = my_ldflags
)
bld.add_group()
# use the above generated binary file
P.S This might seem fairly trivial, but I come from make background, and new to
waf !
Thanks.
--EDIT--
I am able to build the my_binary here, but I want to get its abs path, and reference it in the further steps
build/${build_target}/${your_binary} - unless you overwrite some default value
Update#1
A cut down thing that should keep you going, especially the derival of build targets, also be sure to check the waf book which includes a lot of examples.
def configure(ctx):
ctx.load(...)
ctx.env.appname = APPNAME
ctx.env.version = VERSION
ctx.define(...)
ctx.check_cc(...)
ctx.setenv('debug', env=ctx.env.derive())
ctx.env.CFLAGS = ['-ggdb', '-Wall']
ctx.define('DEBUG',1)
ctx.setenv('release', env=ctx.env.derive())
ctx.env.CFLAGS = ['-O2', '-Wall']
ctx.define('RELEASE',1)
def build(bld):
### subdirs :) under build are usually related to build variant or command
print (">>>>> "+bld.cmd)
print (">>>>> "+bld.variant)
bin = bld.program(...)
from waflib.Build import BuildContext
class release(BuildContext):
cmd = 'release'
variant = 'release'
class debug(BuildContext):
cmd = 'debug'
variant = 'debug'
I noticed that QFileDialog instance is returning absolute paths for the member function selectedFile() that have the wrong separator for the given operating system. This is not expected on a cross platform language (python)
What should I do to correct this so that the rest of my properly OS-independant python code using 'os.sep' can be correct? I don't want to have to remember where I can and can't use it.
You use the os.path.abspath function:
>>> import os
>>> os.path.abspath('C:/foo/bar')
'C:\\foo\\bar'
The answer came from another thread ( HERE ) that mentioned I need to use QDir.toNativeSeparators()
so I did the following in my loop (which should probably be done in pyqt itself for us):
def get_files_to_add(some_directory):
addq = QFileDialog()
addq.setFileMode(QFileDialog.ExistingFiles)
addq.setDirectory(some_directory)
addq.setFilter(QDir.Files)
addq.setAcceptMode(QFileDialog.AcceptOpen)
new_files = list()
if addq.exec_() == QDialog.Accepted:
for horrible_name in addq.selectedFiles():
### CONVERSION HERE ###
temp = str(QDir.toNativeSeparators(horrible_name)
###
# temp is now as the os module expects it to be
# let's strip off the path and the extension
no_path = temp.rsplit(os.sep,1)[1]
no_ext = no_path.split(".")[0]
#... do some magic with the file name that has had path stripped and extension stripped
new_files.append(no_ext)
pass
pass
else:
#not loading anything
pass
return new_files
Background
I've grown tired of the issue with pylint not being able to import files when you use namespace packages and divide your code-base into separate folders. As such I started digging into the astNG source-code which has been identified as the source of the trouble (see bugreport 8796 on astng). At the heart of the issue seems to be the use of pythons own imp.find_module in the process of finding imports.
What happens is that the import's first (sub)package - a in import a.b.c - is fed to find_module with a None path. Whatever path comes back is then fed into find_module the next pass in the look up loop where you try to find b in the previous example.
Pseudo-code from logilab.common.modutils:
path = None
while import_as_list:
try:
_, found_path, etc = find_module(import_as_list[0], path)
#exception handling and checking for a better version in the .egg files
path = [found_path]
import_as_list.pop(0)
The Problem
This is what's broken: you only get the first best hit from find_module, which may or may not have your subpackages in it. If you DON'T find the subpackages, you have no way to back out and try the next one.
I tried explicitly using sys.path instead of None, so that the result could be removed from the path list and a second attempt be made, but python's module finder is clever enough that there doesn't have to be an exact match in the paths, making this approach unusable - to the best of my knowledge anyway.
Teary-eyed Plea
Is there an alternative to find_modules which will return ALL possible matches or take an exclude list? I'm also open to completely different solutions. Preferably not patching python by hand, but it wouldn't be impossible - at least for a local solution.
(Caveat emptor: I'm running python 2.6 and for reasons of current company policy can't upgrade, suggestions for p3k etc won't get marked as accepted unless it's the only answer.)
Since Python 2.5, the right way to do this is with pkgutil.iter_modules() (for a flat list) or pkgutil.walk_packages() (for a subpackage tree). Both are fully compatible with namespace packages.
For example, if I wanted to find just the subpackages/submodules of 'jmb', I would do:
import jmb, pkgutil
for (module_loader, name, ispkg) in pkgutil.iter_modules(jmb.__path__, 'jmb.'):
# 'name' will be 'jmb.foo', 'jmb.bar', etc.
# 'ispkg' will be true if 'jmb.foo' is a package, false if it's a module
You can also use iter_modules or walk_packages to walk all the modules on sys.path; see the docs linked above for details.
I've grown tired of this limitation in PyLint too.
I don't know a replacement for imp.find_modules(), but I think I found another way to deal with namespace packages in PyLint. See my comment on the bug report you linked to (http://www.logilab.org/ticket/8796).
The idea is to use pkg_resources to find namespace packages. Here's my addition to logilab.common.modutils._module_file(), just after while modpath:
while modpath:
if modpath[0] in pkg_resources._namespace_packages and len(modpath) > 1:
module = sys.modules[modpath.pop(0)]
path = module.__path__
This not very refined and only handles top-level namespace packages though.
warning + disclaimer: not tested yet!
before:
for part in parts:
modpath.append(part)
curname = '.'.join(modpath)
# ...
if module is None:
mp_file, mp_filename, mp_desc = imp.find_module(part, path)
module = imp.load_module(curname, mp_file, mp_filename, mp_desc)
after: - thanks pjeby for mentioning pkgutil!
for part in parts:
modpath.append(part)
curname = '.'.join(modpath)
# ...
if module is None:
# + https://stackoverflow.com/a/14820895/611007
# # mp_file, mp_filename, mp_desc = imp.find_module(part, path)
# # module = imp.load_module(curname, mp_file, mp_filename, mp_desc)
import pkgutil
mp_file = None
for loadr,name,ispkg in pkgutil.iter_modules(path=path,prefix='.'.join(modpath[:-1])+'.'):
if name.split('.')[-1] == part:
if not hasattr(loadr,'path') and hasattr(loadr,'archive'):
# with zips `name` was like '.somemodule'
# it gives `RuntimeWarning: Parent module '' not found while handling absolute import`
# I expect the name I need to be 'somemodule'
# TODO: I don't know why python does this or what the correct usage is.
# https://stackoverflow.com/questions/2267984/
if name and name[0] == '.':
name = name[1:]
ldr= loadr.find_module(name,loadr.archive)
module = ldr.load_module(name)
break
imploader= loadr.find_module(name,loadr.path)
mp_file,mp_filename,mp_desc= imploader.file,imploader.filename,imploader.etc
module = imploader.load_module(imploader.fullname)
break
if module is None:
raise ImportError