I am writing a utility in python that needs to check for (and if necessary, install and even upgrade) various other modules with in a target project/virtualenv, based on user supplied flags and/or input. I am currently trying to utilize 'pip' directly/programatically (because of it's existing support for the various repo types I will need to access), but I am having difficulty in finding examples or documentation on using it this way.
This seemed like the direction to go:
import pip
vcs = pip.vcs.VersionControl(url="http://path/to/repo/")
...but it gives no joy.
I need help with some of the basics aparently - like how can I use pip to pull/export a copy of an svn repo into a given local directory. Ultimately, I will also need to use it for git and mercurial checkouts as well as standard pypi installs. Any links, docs or pointers would be much appreciated.
Pip uses a particular format for vcs urls. The format is
vcsname+url#rev
#rev is optional, you can use it to reference a specific commit/tag
To use pip to retrieve a repository from a generic vcs to a local directory you may do this
from pip.vcs import VcsSupport
req_url = 'git+git://url/repo'
dest_path = '/this/is/the/destination'
vcs = VcsSupport()
vc_type, url = req_url.split('+',1)
backend = vcs.get_backend(vc_type)
if backend:
vcs_backend = backend(req_url)
vcs_backend.obtain(dest_path)
else:
print('Not a repository')
Check https://pip.pypa.io/en/stable/reference/pip_install/#id8 to know which vcs are supported
Related
Pip has a configuration which is typically in ~/.pip/pip.conf on Linux, %APPDATA%\pip\pip.ini on Windows, and possibly other locations on virtual environments.
I could write some code to locate Pip's config file and then parse it using an ini-file parser (included with Python), however it occurs to me that this code must already exist within Pip. Pip surely must have a mechanism to locate and parse it's own configuration file.
I'd like to be able to access that configuration via Pip's API. In particular I'm trying to get hold of the index URL that Pip is using (along with any credentials which may be embedding). That will allow my service to guarantee that it's going to hit the same repository that Pip used to install from.
Is there an easy way to access this information?
The objective here is to access Pip's configuration information without having to re-implement the code which searches for Pip's config file.
There's no good way to do this, but this gets you somewhat directly to the user's config file. This won't work for site-configs.
import os
import pip.appdirs
import pip.locations
os.path.join(pip.appdirs.user_config_dir("pip"), pip.locations.config_basename)
And a way that gets all the config file locations:
>>> from pip.baseparser import ConfigOptionParser
>>> ConfigOptionParser(name="foo").get_config_files()
['C:\\ProgramData\\pip\\pip.ini', 'C:\\Users\\salimfadhley\\pip\\pip.ini', 'C:\\Users\\salimfadhley\\AppData\\Roaming\\pip\\pip.ini']
>>>
I want to list all the projects using Mercurial with Python.
I downloaded the package "import hglib" but I did not find in the documentation, the functions, etc... which could help me. Somebody is would know how can it be done?
PS: i found some informations in this links:
- https://www.mercurial-scm.org/wiki/MercurialApi
- http://pythonhosted.org/hgapi/index.html#hgapi.hgapi.Repo.command
but it wasn't what i was looking for...
So, what I Understood from your question is probably you want to list the nested repositories inside a mercurial repo. In much more easier terms you must be working with a mercurial forest and you want to list the sub repos deep under and it seems that there is a python package named hgnested written to deal with mercurial forests.
I took a patch of code and played with it to meet what we want and this is what I came up with.
from mercurial import hg, ui
import hgnested
def getNestedRepos(ui, source, **opts):
origsource = ui.expandpath(source)
remotesource, remotebranch =hg.parseurl(origsource,opts.get('branch'))
if hasattr(hg, 'peer'):
remoterepo = hg.peer(ui, opts, remotesource)
localrepo = remoterepo.local()
if localrepo:
remoterepo = localrepo
else:
remoterepo = hg.repository(hg.remoteui(ui, opts), remotesource)
return remoterepo.nested
print getNestedRepos(ui.ui(), <path to mercurial forest>)
But there is another scrapy way where you need to visit all the sub directories recursively and look for the presence of .hg file.
Note: If you want to list repos from a remote repository make sure hgnested package is installed in that remote server and the repo path you are passing is a mercurial forest head.
Q1:
I have an Artifactory PyPi enabled repo my-pypi-repo where I can publish my packages. When uploading via python setup.py -sdist, I get a structure like this:
my-pypi-repo|
|my_package|
|x.y.z|
|my_package-x.y.z.tar.gz
The problem is this structure will not match any "allowed" repo layout in Artifactory, since [org] or [orgPath] are mandatory:
Pattern '[module]/[baseRev]/[module]-[baseRev].[ext]' must at-least
contain the tokens 'module', 'baseRev' and 'org' or 'orgPath'.
I managed to publish to a path by 'hacking' the package name to myorg/my_package, but then pip cannot find it, so it's pretty useles.
Q2:
Has anyone tried the "ci-repo" and "releases-repo" with promotion for Python using Artifactory?
What I would like to achieve:
CI repo:
my_package-1.2.3+build90.tar.gz When this artifact gets promoted build metadata gets dropped
Releases repo:
my_package-1.2.3.tar.gz
I can achieve this via repo layouts (providing I resolve Q1). The problem is how to deal with the "embedded" version inside my Python script, hardcoded in setup.py.
I'd rather not rebuild the package again, for best practices.
I am running into the same issue in regards to your first question/problem. When configuring my system to publish to artifactory using pip, it uses the format you described.
As you mentioned, the [org] or [orgPath] is mandatory and this basically breaks all the REST API functionality, like searching for latest version, etc. I'm currently using this as my Artifact Path Pattern:
[org]/[module]/[baseRev].([fileItegRev])/[module]-[baseRev].([fileItegRev]).[ext]
The problem is that pip doesn't understand the concept of [org] in this case. I'm temporarily using a python script to publish my packages to Artifactory to get around this. Hopefully this is something that can be addressed by the jFrog team.
The python script simply uses Artifactory's REST API to publish to my local pypi repository, tacking on a few properties so that some of the REST API functions work properly, like Artifact Latest Version Search Based on Properties.
I need to be able to use that call because we're using Chef in-house and we use that method to get the latest version. The pypi.version property that gets added when publishing via python setup.py sdist upload -r local doesn't work with the REST API so I have to manually add the version property. Painful to be honest since we can't add properties when using the upload option with setup.py. Ideally I'd like to be able to do everything using pip, but at the moment this isn't possible.
I'm using the requests package and the upload method in the Artifactory documentation here. Here is the function I'm using to publish adding a few properties (feel free to add more if you need):
def _publish_artifact(name, version, path, summary):
base_url = 'http://server:8081/artifactory/{0}'.format(PYPI_REPOSITORY)
properties = ';version={0};pypi.name={1};pypi.version={0};pypi.summary={2}'\
.format(version, name, summary)
url_path = '/Company/{0}/{1}/{0}-{1}.zip'.format(name, version)
url = '{0}{1}{2}'.format(base_url, properties, url_path)
dist_file = r'{0}\dist\{1}-{2}.zip'.format(path, name, version)
files = {'upload_file': open(dist_file, 'rb')}
s = requests.Session()
s.auth = ('username', 'password')
reply = s.put(url, files=files)
logger.info('HTTP reply: {0}'.format(reply))
A1: Artifactory layouts aren't enforcive, you can deploy any file under any path to any repo. Some layout-related features, like snapshots cleanup won't work then, but I don't think you need them anyway.
A2: The best solution will be to code your promotion in a promotion user plugin. Renaming artifacts on the fly during their promotion to another repo is one of the most popular scenarios of this kind of plugin.
My context is appengine_config.py, but this is really a general Python question.
Given that we've cloned a repo of an app that has an empty directory lib in it, and that we populate lib with packages by using the command pip install -r requirements.txt --target lib, then:
dirname ='lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)
For importing purposes, we can add such a filesystem path to the beginning of the Python path in the following way (we use index 1 because the first position should remain '.', the current directory):
sys.path.insert(1, dirpath)
However, that won't work if any of the packages in that directory are namespace packages.
To support namespace packages we can instead use:
site.addsitedir(dirpath)
But that appends the new directory to the end of the path, which we don't want in case we need to override a platform-supplied package (such as WebOb) with a newer version.
The solution I have so far is this bit of code which I'd really like to simplify:
sys.path, remainder = sys.path[:1], sys.path[1:]
site.addsitedir(dirpath)
sys.path.extend(remainder)
Is there a cleaner or more Pythonic way of accomplishing this?
For this answer I assume you know how to use setuptools and setup.py.
Assuming you would like to use the standard setuptools workflow for development, I recommend using this code snipped in your appengine_config.py:
import os
import sys
if os.environ.get('CURRENT_VERSION_ID') == 'testbed-version':
# If we are unittesting, fake the non-existence of appengine_config.
# The error message of the import error is handled by gae and must
# exactly match the proper string.
raise ImportError('No module named appengine_config')
# Imports are done relative because Google app engine prohibits
# absolute imports.
lib_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'libs')
# Add every library to sys.path.
if os.path.isdir(lib_dir):
for lib in os.listdir(lib_dir):
if lib.endswith('.egg'):
lib = os.path.join(lib_dir, lib)
# Insert to override default libraries such as webob 1.1.1.
sys.path.insert(0, lib)
And this piece of code in setup.cfg:
[develop]
install-dir = libs
always-copy = true
If you type python setup.py develop, the libraries are downloaded as eggs in the libs directory. appengine_config inserts them to your path.
We use this at work to include webob==1.3.1 and internal packages which are all namespaced using our company namespace.
You may want to have a look at the answers in the Stack Overflow thread, "How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)," but for your particular predicament with namespace packages, you're running up against a long-standing issue I filed against site.addsitedir's behavior of appending to sys.path instead of inserting after the first element. Please feel free to add to that discussion with a link to this use case.
I do want to address something else that you said that I think is misleading:
My context is appengine_config.py, but this is really a general Python
question.
The question actually arises from the limitations of Google App Engine and the inability to install third-party packages, and hence, seeking a workaround. Rather than manually adjusting sys.path and using site.addsitedir. In general Python development, if your code uses these, you're Doing It Wrong.
The Python Packaging Authority (PyPA) describes the best practices to put third party libraries on your path, which I outline below:
Create a virtualenv
Mark out your dependencies in your setup.py and/or requirements files (see PyPA's "Concepts and Analyses")
Install your dependencies into the virtualenv with pip
Install your project, itself, into the virtualenv with pip and the -e/--editable flag.
Unfortunately, Google App Engine is incompatible with virtualenv and with pip. GAE chose to block this toolset in an attempt sandbox the environment. Hence, one must use hacks to work around the limitations of GAE to use additional or newer third party libraries.
If you dislike this limitation and want to use standard Python tooling for managing third-party package dependencies, other Platform as a Service providers out there eagerly await your business.
I'm using suds (brilliant library, btw), and I'd like to make it portable (so that everyone who uses the code that relies on it, can just checkout the files and run it).
I have tracked down 'suds-0.4-py2.6.egg' (in python/lib/site-packages), and put it in with my files, and I've tried:
import path.to.egg.file.suds
from path.to.egg.file.suds import *
import path.to.egg.file.suds-0.4-py2.6
The first two complain that suds doesn't exist, and the last one has invalid syntax.
In the __init__.py file, I have:
__all__ = [ "FileOne" ,
"FileTwo",
"suds-0.4-py2.6"]
and have previously tried
__all__ = [ "FileOne" ,
"FileTwo",
"suds"]
but neither work.
Is this the right way of going about it? If so, how can I get my imports to work. If not, how else can I achieve the same result?
Thanks
You must add your egg file to sys.path, like this:
import sys
# insert at 0 instead of appending to end to take precedence
# over system-installed suds (if there is one).
sys.path.insert(0, "suds-0.4-py2.6.egg")
import suds
.egg files are zipped archives; hence you cannot directly import them as you have discovered.
The easy way is to simply unzip the archive, and then copy the suds directory to your application's source code directory. Since Python will stop at the first module it discovers; your local copy of suds will be used even if it is not installed globally for Python.
One step up from that, is to add the egg to your path by appending it to sys.path.
However, the proper way would be to package your application for distribution; or provide a requirements file that lets other people know what external packages your program depends on.
Usually I distribute my program with a requirements.txt file that contain all dependencies and their version.
The users can then install these libraries with:
pip install -r requirements.txt
I don't think including eggs with your code is a good idea, what if the user use python2.7 instead of python2.6
More info about requirement file: http://www.pip-installer.org/en/latest/requirements.html