Since upgrading to subversion 1.7 I get "unrecognized .svn/entries format" when running buildout. I notice there is an unresolved bug reports for both distribute and setuptools for this error and it also seems that you can use setuptools_subversion to resolve the issue.
What I can't find out how to do is install setuptools_subversion so that buildout picks it up. Please can someone help?
I've tried
downloading it and running python setup.py install
adding it to the eggs list of the [buildout] part of my buildout configuration
You need to install it at the python site-packages level; easy_install (used under the hood by buildout) needs it available before it'll install anything else.
That said, the python setup.py install stanza should have installed it just fine; check by running the following test:
$ python -m setuptools_subversion
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/setuptools_subversion.py directory
That should print the installation path of the module, like it did for me in the above example. You could try to use pip or easy_install for automatic download:
$ pip install setuptools_subversion
or
$ easy_install setuptools_subversion
You can do that in a virtualenv if you want to isolate the installation. Because this is basically a dependency for svn 1.7, installing this at the same level as the svn binary (usually system wide) is certainly acceptable and the norm.
Note that the unrecognized .svn/entries format error message will not disappear, but your buildout will otherwise succeed. The message is printed no matter what as easy_install first tries the internal .svn parser before deferring to the external plugin.
If you really, really want to verify if the plugin is installed, run the following python code:
import pkg_resources
for entrypoint in pkg_resources.iter_entry_points('setuptools.file_finders'):
print entrypoint
On my system this prints:
svn = setuptools_subversion:listfiles
svn_cvs = setuptools.command.sdist:_default_revctrl
git = setuptools_git:gitlsfiles
hg = setuptools_hg:hg_file_finder
Related
I've got a couple of projects here for which I'm preparing documentation at the moment, hosted at readthedocs.org. FYI, all of them use poetry and I use custom .readthedocs.yml files with this entry:
python:
install:
- method: pip
path: .
It works fine for most projects, but it fails for two for different reasons during installation of the project via pip:
The first one uses PyGObject, which failes like this:
Package gobject-introspection-1.0 was not found in the pkg-config search path.
Perhaps you should add the directory containing `gobject-introspection-1.0.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gobject-introspection-1.0' found
Command '('pkg-config', '--print-errors', '--exists', 'gobject-introspection-1.0 >= 1.56.0')' returned non-zero exit status 1.
Try installing it with: 'sudo apt install libgirepository1.0-dev'
So it seems that PyGObject cannot be installed without some system packages to be installed. I could rearrange the code so that the import is not top-level. But still I need it in the dependencies. Can I tell pip install to ignore this single package somehow? Any other idea?
The second project compiles some C++ code via Cython and fails, because it's missing a library. I use a custom build script in the pyproject.toml:
[tool.poetry.build]
script = "build.py"
generate-setup-file = false
Is there some flag in pip that I could set and retrieve in build.py to skip the compilation? Or is there a better way?
What I should have:
I want my Yocto Project to build a package for my Python project with all dependencies inside. The project has to run out of box on the resulting read-only sdcard image.
It simply should install all requirements in the required version to the package.
What I tried without luck:
Calling pip in do_install():
"pip/pip3 is not found", even it's in RDEPENDS.
Anyway, I really prefer this way.
With inherit pypi:
When trying with inherit pypi, it tries to get also my local sources (my pyton project) from pypi. And I have always to copy the requirements to the recipe. This is not my preferred way.
Calling pip in pkg_postinst():
It tries to install the modules on first start and fails, because the system has no internet connection and it's a read-only system. It must run out of the box without installation on first boot time. Does its stuff to late.
Where I'll get around:
There should be no need to change anything in the recipes when something changes in requirements.txt.
Background information
I'm working with Yocto Rocko in a Linux environment.
In the Hostsystem, there is no pip installed. I want to run this one installed from RDEPENDS in the target system.
Building the Package (only this recipe) with:
bitbake myproject
Building the whole sdcard image:
bitbake myProject-image-base
The recipe:
myproject.bb (relevant lines):
RDEPENDS_${PN} = "python3 python3-pip"
APP_SOURCES_DIR := "${#os.path.abspath(os.path.dirname(d.getVar('FILE', True)) + '/../../../../app-sources')}"
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI = " \
file://${APP_SOURCES_DIR}/myProject \
...
"
inherit allarch # tried also with pypi and setuptools3 for the pypi way.
do_install() { # Line 116
install -d -m 0755 ${D}/myProject
cp -R --no-dereference --preserve=mode,links -v ${APP_SOURCES_DIR}/myProject/* ${D}/myProject/
pip3 install -r ${APP_SOURCES_DIR}/myProject/requirements.txt
# Tried also python ${APP_SOURCES_DIR}/myProject/setup.py install
}
# Tried also this, but it's no option because the data MUST be included in the Package:
# pkg_postinst_${PN}() {
# #!/bin/sh -e
# pip3 install -r /myProject/requirements.txt
# }
FILES_${PN} = "/myProject/*"
Resulting Errors:
Expected to install the listed modules from requirements.txt into the myProject package, so that the python app will run directly on the resulting readonly sdcard image.
With pip, I get:
| /*/tmp/work/*/myProject/0.1.0-r0/temp/run.do_install: 116: pip3: not found
| WARNING: exit code 127 from a shell command.
| ERROR: Function failed: do_install ...
When using pypi:
404 Not Found
ERROR: myProject-0.1.0-r0 do_fetch: Fetcher failure for URL: 'https://files.pythonhosted.org/packages/source/m/myproject/myproject-0.1.0.tar.gz'. Unable to fetch URL from any source.
=> But it should not fetch myProject, since it is already local and nowhere remote.
Any ideas? What would be the best way to reach to a ready to use sdcard image without the need to change recipes when requirements.txt changes?
You should use RDEPENDS_${PN} to take care of your dependencies for your app in the recipe.
For example, assuming your python app needs aws-iot-device-sdk-python module, you should add it to RDEPENDS in the recipe. In your case, it would be like this:
RDEPENDS_${PN} = "python3 \
python3-pip \
python3-aws-iot-device-sdk-python \
"
Here's the link showing the Python modules supported by OpenEmbedded Layer.
https://layers.openembedded.org/layerindex/branch/master/layer/meta-python/
If the modules you need are not there, you will likely need to create recipes for the modules.
My newest findings:
Yocto/bitbake seems to suppress interpreting the requirements, because this breaks automatic dependency resolving what could lead to conflicts.
Reason: The required modules from setup.py would not be stored as independent packages, but as part of my package. So, bitbake does not know about this modules what could conflict with other packages that probably requires same modules in different versions.
What was in my recipe:
MY_INSTALL_ARGS = "--root=${D} \
--prefix=${prefix} \
--install-lib=${PYTHON_SITEPACKAGES_DIR} \
--install-data=${datadir}"
do_install() {
PYTHONPATH=${PYTHON_SITEPACKAGES_DIR} \
${STAGING_BINDIR_NATIVE}/${PYTHON_PN}-native/${PYTHON_PN} setup.py install ${MY_INSTALL_ARGS}
}
If I execute this outside of bitbake as python3 setup.py install ${MY_INSTALL_ARGS}, all will be installed correctly, but in the recipe, no requirements are installed.
There is a parameter --no-deps, but I didn't find where it is set.
I think there could be one possibility to exploit the requirements out of setup.py:
Find out where to disable --no-deps in the openembedded/poky layer for easy_install.
Creating a separate PYTHON_SITEPACKAGES_DIR
Install this separate PYTHON_SITEPACKAGES_DIR in eg the home directory as private python modules dir.
This way, no python module would trigger a conflict.
Since I do not have the time to experiment with this, I'll define now one recipe per requirement.
You try installing pip?
Debian
apt-get install python-pip
apt-get install python3-pip
Centos
yum install python-pip
Edit: Yes I know this question already exists, except my question is a bit different and none of the solutions fixed it.
I do most of my Python stuff when I'm at work and not on my personal machine, but I decided to install it on my personal computer as well. I fresh installed python 3.6.1, and created a virtual environment with virtualenv. Then within the virtualenv I tried to pip install urllib (or any module) and I received the error:
(pdbot) C:\Users\user\Documents\pdbot>pip install urllib
Collecting urllib
Using cached urllib-1.21.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\user\AppData\Local\Temp\pip-build-50tn0wlb\urllib\setup.py", line 191
s.connect((base64.b64decode(rip), 017620))
^
SyntaxError: invalid token
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\Users\user\AppData\Local\Temp\pip-build-50tn0wlb\urllib\
I read elsewhere that this error had something to do with setuptools not being properly installed. So I ran this to attempt to fix the issue:
easy_install -U setuptools
I ended up receiving an even weirder error next:
(pdbot) C:\Users\zeke\Documents\pdbot>easy_install -U setuptools
Searching for setuptools
Reading https://pypi.python.org/simple/setuptools/
Downloading https://pypi.python.org/packages/a9/23/720c7558ba6ad3e0f5ad01e0d6ea2288b486da32f053c73e259f7c392042/setuptools-36.0.1.zip#md5=430eb106788183eefe9f444a300007f0
Best match: setuptools 36.0.1
Processing setuptools-36.0.1.zip
Writing C:\Users\zeke\AppData\Local\Temp\easy_install-jhg1val_\setuptools-36.0.1\setup.cfg
Running setuptools-36.0.1\setup.py -q bdist_egg --dist-dir C:\Users\zeke\AppData\Local\Temp\easy_install-jhg1val_\setuptools-36.0.1\egg-dist-tmp-8apak7kn
warning: no files found matching '*' under directory 'setuptools\_vendor'
Copying setuptools-36.0.1-py3.6.egg to c:\users\zeke\documents\pdbot\lib\site-packages
Adding setuptools 36.0.1 to easy-install.pth file
Installing easy_install-script.py script to c:\users\zeke\documents\pdbot\Scripts
Installing easy_install.exe script to c:\users\zeke\documents\pdbot\Scripts
error: [WinError 5] Access is denied: 'c:\\users\\zeke\\documents\\pdbot\\Scripts\\easy_install.exe'
This looks like a permissions error, but I ran these both in an administrator command prompt (Windows 10) and got the same result. I am the only user on this computer and I have all admin permissions. Is this virtualenv causing an issue? How do I remedy it?
EDIT: I was able to fix the permissions issue by leveraging the python executable like so:
python -m easy_install -U setuptools
But it didn't fix the python setup.py egg_info issue. I still get this error message when trying to pip install anything:
Command "python setup.py egg_info" failed with error code 1 in C:\Users\user\AppData\Local\Temp\pip-build-50tn0wlb\urllib\
I have tried both python -m pip install urllib and pip install urllib and neither work.
I had the same problem when trying to install urllib, but after doing a pip search urllib, I discovered that the problem was due to the version of urllib. From the search:
$ pip search urllib
...
> urllib5 (5.0.0) - Just increment the number and create a new lib. Never fix the original one.
At the end, a simple
pip install urllib5
within an elevated shell solved it.
Your problem has to do with permissions. The related/similar tools setup_tools, easy_install, and pip all tend to set a default set of permissions on files and folders they try to create in the package installation folder(s), rather than trying to match access permissions of the location they're installing in.
On Linux systems, where files and folders individually have permissions, this is frequently bypassed with the sudo command. On Windows, the equivalent is to run the installer as an Administrator. Since you're in the console, you have to open a console with Administrator privileges to run the pip command in.
Notable under Windows, the modules installed with pip from an Administrator console are still accessible to all users of the system that have the proper path in the PYTHONPATH system environment variable. Under Linux however, the problem is exacerbated by the fact that the files themselves may not be created with read and execute access for other users and may need to have their permissions manually modified after installation.
WARNING: urllib vs urllib2 vs urllibx
Both other answers claim that the problem is you're not specifying the correct "version" of the module in the call to pip. Neither is correct, as the error clearly indicates an installation folder access permissions violation causing the failure, but they also incorrectly recommended VERY unsafe behavior.
pip install urllib != pip install urllib5 these are two completely different packages.
The documentation for pip (https://packaging.python.org/tutorials/installing-packages/#id17) clearly says the way to specify a module version explicitly is pip install 'urllib==5'.
As part of how the package management engine implemented by pip works, running the command pip install urllib will always try to use the latest version of the urllib package, so you shouldn't need to specify the version unless you have some reason that you need a very specific version of the module.
There are two points to make in order to answer your question:
1. You are lucky you did not install that package!
The package you were trying to install was a maliciously created python package that was designed to look like a real package (in this case urllib3). If you had installed it, the package would have operated as normal except it would have sent some basic information about the system on which you installed the package to a URL (you can see more details on this here). You can read more about this fake package at either of the following links:
https://app.threatconnect.com/auth/incident/incident.xhtml?incident=5256822&owner=Common%20Community (you can sign up for a free account to view this one)
http://www.nbu.gov.sk/skcsirt-sa-20170909-pypi/index.html
Sending basic information about your systems to an unknown source isn't the worst thing you could do, but is certainly something you want to avoid when possible.
2. To properly install a package...
Specifically urllib:
To install urllib, you need to specify the version of the package you would like to install. For example, pip install urllib3.
Any package in general:
As #Elisabete Coelho suggested, you can use the pip search <package-name> feature to view the available packages. This is not perfect, however, as it may list malicious libraries like the one you were trying to install. A good guideline is that you should follow the installation instructions in a package's documentation closely to avoid any unforeseen issues. This is just an unfortunate necessity of living in a world where people make pretend python packages.
Installing MySQLDb (Python module) without Internet Connection
Hi all,
I need to install MySQLDb on a SLES 11 Development Server. This Dev Server does not have access to the public internet due to corporate firewall policies. I was assuming this would merely be a nuisance which would force me to do source installations etc on things more easily installed otherwise. Instead, I have hit a wall, during my attempt to ultimately install and run Django with MySQL support.
I have downloaded and unzipped MySQL-python-1.2.4, and I am attempting to run its setup.py. However, any attempt to run setup.py, even just sudo python setup.py --help, reults in the following
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.28.tar.gz
And then of course the download fails.
I downloaded and installed the current version of the distribute module (0.7.3), via sudo python setup.py build install. I assumed this would keep mysql-python's setup.py from trying to download distribute. But that does not appear to matter. I did have a quick look at mysql-client's setup.py to see where the download was being forced. It appears that its setup.py does this:
from distribute_setup import use_setuptools
use_setuptools()
Which calls this:
def use_setuptools(version=DEFAULT_VERSION, download_base=DEFAULT_URL,
to_dir=os.curdir, download_delay=15, no_fake=True):
# making sure we use the absolute path
to_dir = os.path.abspath(to_dir)
was_imported = 'pkg_resources' in sys.modules or \
'setuptools' in sys.modules
try:
try:
import pkg_resources
if not hasattr(pkg_resources, '_distribute'):
if not no_fake:
_fake_setuptools()
raise ImportError
except ImportError:
return _do_download(version, download_base, to_dir, download_delay)
And this, the import of pkg_resources fails (I have reproduced this from the command line); and the exception handler tries a download, which of course fails.
My understanding is that distribute is deprecated anyway, and that setuptools should be used instead. I do have setuptools installed; but is the mysqldb module hardcoded to use distribute, and possibly a specific version of distribute, and that's my issue? To be honest at this point I'm a bit confused about modules, dependencies, etc in Python (I'm quite mediocre at Python).
Thanks all,
Bean
git clone https://github.com/PyMySQL/PyMySQL or download tarball. for mysqldb: https://github.com/farcepest/MySQLdb1
untar it
run sudo python setup.py install
That's all.
Guess it's too late, but for the sake of the future googlers...
I had the same problem. To solve it I had to comment these lines:
if not hasattr(pkg_resources, '_distribute'):
if not no_fake:
_fake_setuptools()
raise ImportError
After this I was able to install MySQL-python via python setup.py install.
The answer to this question appears to be version dependent. MySQLDb version 1.2.5 (newest version as of this writing) and later do not require distribute so python setup.py install will work. Version 1.2.5 was released 01/2014 so this question is just that old.
I have installed bottle on my Ubuntu Linux server using
sudo pip install bottle
and it is installed to: /usr/local/lib/python2.7/dist-packages
But I also have Python 3.2 installed on my system, and I want to access bottle from Python 3.2. Python 3.2 does not seem to recognise that bottle is installed.
What am I doing wrong?
You'd have to separately install it for Python 3.2 (with e.g. sudo pip-3.2 install bottle).
It's currently in python2.7/dist-packages, meaning that only 2.7 is going to load it. You could try to add that to your PYTHONPATH or similar, but that will very rarely work between Python 2 and 3 because the source files aren't quite compatible. (Any C extensions are also certainly not going to work, though bottle doesn't have any of those.)
Unfortunately, although that command works, it looks like the version of bottle in pypi isn't Python 3-compatible even when installed through pip-3.2:
In [1]: import bottle
File "/Library/Frameworks/Python.framework/Versions/3.2/bin/bottle.py", line 373
except re.error, e:
^
SyntaxError: invalid syntax
The homepage claims that it works with 3.x, but I got that error installing with both pip and easy_install. The latest development version, which is just a single file linked from the homepage, seems to work, though.
You are not doing anything wrong. Pip uses the /usr/bin/python by default and only installs there.
Unless you want to setup virtualenv-s, you probably best copy the current pip to pip3.2 and edit that to call python 3.2:
sudo -s -H
p=$(which pip)
cat $p | sed "1s|/usr/bin/python|$(which python3.2)|" > $p"3.2"
chmod 755 $p"3.2"
exit
You now have a pip3.2 that will install bottle so python3.2 can use it. If you get an error running pip3.2 about not finding pkg_resources look at No module named pkg_resources
I'll answer this myself. Turns out the latest release version of pip does not include pip-3.2. You need to download the development version and use that, which includes pip-3.2.
Just download it manually from offsite. It is just one file.
Place it into the lib/site-packages folder and give the file proper rights.