With my Java projects at present, I have full version control by declaring it as a Maven project. However I now have a Python project that I'm about to tag 0.2.0 which has no version control. Therefore should I come accross this code at a later date, I won't no what version it is.
How do I add version control to a Python project, in the same way Maven does it for Java?
First, maven is a build tool and has nothing to do with version control. You don't need a build tool with Python -- there's nothing to "build".
Some folks like to create .egg files for distribution. It's as close to a "build" as you get with Python. This is a simple setup.py file.
You can use SVN keyword replacement in your source like this. Remember to enable keyword replacement for the modules that will have this.
__version__ = "$Revision$"
That will assure that the version or revision strings are forced into your source by SVN.
You should also include version keywords in your setup.py file.
Create a distutils setup.py file. This is the Python equivalent to maven pom.xml, it looks something like this:
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
If you want dependency management like maven, take a look at setuptools.
Ants's answer is correct, but I would like to add that your modules can define a __version__ variable, according to PEP 8, which can be populated manually or via Subversion or CVS, e.g. if you have a module thingy, with a file thingy/__init__.py:
___version___ = '0.2.0'
You can then import this version in setup.py:
from distutils.core import setup
import thingy
setup(name='thingy',
version=thingy.__version__,
py_modules=['thingy'],
)
Related
It seems from here and here, that setuptools does not work properly when a project has a module named parser in it. I am trying to write a basic setup script for this project: https://github.com/karlmoritz/bolinas . This is not my repository and I want to avoid making any changes to this code (or if it is not possible to make no changes, then only making minimal changes).
As you can see, there is a package called parser that conflicts with the default python installation. This is my setup.py:
from setuptools import setup, find_packages
setup(
name = "Bolinas",
version = "0.1",
packages = find_packages(),
scripts = ['bolinas.py', 'config.py'],
)
When I run it, I get the same errors as in the links I provided above.
Is there anything I can do to make this work without renaming the module?
These guys really have top level packages called "common", "config" and "parser"? No sane person would install this (or would later rue the day). Its not a problem with setup.py at all. Its a problem with a package that overrides standard system modules like "parser".
As stands, bolinas is not architected to be installable. To do so, its modules should all be moved under a package called 'bolinas' so that the base level namespace isn't littered with a bunch of vaguely named modules.
My context is appengine_config.py, but this is really a general Python question.
Given that we've cloned a repo of an app that has an empty directory lib in it, and that we populate lib with packages by using the command pip install -r requirements.txt --target lib, then:
dirname ='lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)
For importing purposes, we can add such a filesystem path to the beginning of the Python path in the following way (we use index 1 because the first position should remain '.', the current directory):
sys.path.insert(1, dirpath)
However, that won't work if any of the packages in that directory are namespace packages.
To support namespace packages we can instead use:
site.addsitedir(dirpath)
But that appends the new directory to the end of the path, which we don't want in case we need to override a platform-supplied package (such as WebOb) with a newer version.
The solution I have so far is this bit of code which I'd really like to simplify:
sys.path, remainder = sys.path[:1], sys.path[1:]
site.addsitedir(dirpath)
sys.path.extend(remainder)
Is there a cleaner or more Pythonic way of accomplishing this?
For this answer I assume you know how to use setuptools and setup.py.
Assuming you would like to use the standard setuptools workflow for development, I recommend using this code snipped in your appengine_config.py:
import os
import sys
if os.environ.get('CURRENT_VERSION_ID') == 'testbed-version':
# If we are unittesting, fake the non-existence of appengine_config.
# The error message of the import error is handled by gae and must
# exactly match the proper string.
raise ImportError('No module named appengine_config')
# Imports are done relative because Google app engine prohibits
# absolute imports.
lib_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'libs')
# Add every library to sys.path.
if os.path.isdir(lib_dir):
for lib in os.listdir(lib_dir):
if lib.endswith('.egg'):
lib = os.path.join(lib_dir, lib)
# Insert to override default libraries such as webob 1.1.1.
sys.path.insert(0, lib)
And this piece of code in setup.cfg:
[develop]
install-dir = libs
always-copy = true
If you type python setup.py develop, the libraries are downloaded as eggs in the libs directory. appengine_config inserts them to your path.
We use this at work to include webob==1.3.1 and internal packages which are all namespaced using our company namespace.
You may want to have a look at the answers in the Stack Overflow thread, "How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)," but for your particular predicament with namespace packages, you're running up against a long-standing issue I filed against site.addsitedir's behavior of appending to sys.path instead of inserting after the first element. Please feel free to add to that discussion with a link to this use case.
I do want to address something else that you said that I think is misleading:
My context is appengine_config.py, but this is really a general Python
question.
The question actually arises from the limitations of Google App Engine and the inability to install third-party packages, and hence, seeking a workaround. Rather than manually adjusting sys.path and using site.addsitedir. In general Python development, if your code uses these, you're Doing It Wrong.
The Python Packaging Authority (PyPA) describes the best practices to put third party libraries on your path, which I outline below:
Create a virtualenv
Mark out your dependencies in your setup.py and/or requirements files (see PyPA's "Concepts and Analyses")
Install your dependencies into the virtualenv with pip
Install your project, itself, into the virtualenv with pip and the -e/--editable flag.
Unfortunately, Google App Engine is incompatible with virtualenv and with pip. GAE chose to block this toolset in an attempt sandbox the environment. Hence, one must use hacks to work around the limitations of GAE to use additional or newer third party libraries.
If you dislike this limitation and want to use standard Python tooling for managing third-party package dependencies, other Platform as a Service providers out there eagerly await your business.
i wrote a little module and i would like to know what are the basic steps to package it in order to send it to pypi:
what is the file hierarchy?
how should i name files?
should i use distutils to create PKG-INFO?
where should i include my documentation (made with sphinx)?
I recommend reading The Hitchhiker's Guide to Packaging. Specifically, you should look at the Quick Start section, which describes how to:
Lay out your project
Describe your project
Create your first release
Register your package with the Python Package Index (PyPI)
Upload your release, then grab your towel and save the Universe!
You should also look at the Current State of Packaging in the Introduction to Packaging section, as this helps untangle some of the confusion surrounding setuptools, distutils, distutils2, and distribute.
Update Re: How to Name Files
The excerpt below is from PEP8, which answers your question about how to name files:
Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.
Since module names are mapped to file names, and some file systems are case insensitive and truncate long names, it is important that module names be chosen to be fairly short -- this won't be a problem on Unix, but it may be a problem when the code is transported to older Mac or Windows versions, or DOS.
an example is always the best way to see how to do:
http://packages.python.org/an_example_pypi_project/
Maybe this CheeseShopTutorial is of help for you. From there:
Submitting Packages to the Package
Index
If you have some Python modules or packages that you would like to
share with the Python community, we'd love to have them included in
the Python Package Index! First, if you haven't done so, you will want
to get your project organized. You might follow the guidelines at
ProjectFileAndDirectoryLayout. After that, you'll want to read the
Python documentation regarding creating distributions:
http://docs.python.org/distutils/index.html.
You can also check Writing a Package in Python by Tarek Ziadé from Tarek's book "Expert Python Programming" where questions about development and distribution are addressed in great detail
Matthew Rankin's answer tells you how to organize your project file heirarchy, but I find myself having to look up the commands to execute every time I want to update a project on PyPI. So here they are:
As described on the PyPi site:
Create a setup.py file (here's an example)
Register a username/password with PyPi
Do this:
python setup.py sdist
python setup.py bdist_wheel
python setup.py register
twine upload dist/*
Look for your package on the PyPi site:
https://pypi.python.org/pypi?%3Aaction=index
On another machine try running:
pip install [your package name]
Most important thing is prepare your setup.py properly. Then:
setup.py sdist bdist_wheel to generate distribution archives to dist/ folder
twine upload dist/* to upload the archives to PyPi (with your PyPi username/password)
Here is an example of setup.py:
from setuptools import setup, find_packages
with open('README.md') as readme_file:
README = readme_file.read()
with open('HISTORY.md') as history_file:
HISTORY = history_file.read()
setup_args = dict(
name='elastictools',
version='0.1.2',
description='Useful tools to work with Elastic stack in Python',
long_description_content_type="text/markdown",
long_description=README + '\n\n' + HISTORY,
license='MIT',
packages=find_packages(),
author='Thuc Nguyen',
author_email='gthuc.nguyen#gmail.com',
keywords=['Elastic', 'ElasticSearch', 'Elastic Stack', 'Python 3', 'Elastic 6'],
url='https://github.com/ncthuc/elastictools',
download_url='https://pypi.org/project/elastictools/'
)
install_requires = [
'elasticsearch>=6.0.0,<7.0.0',
'jinja2'
]
if __name__ == '__main__':
setup(**setup_args, install_requires=install_requires)
You can find more detail tutorial here: https://medium.com/#thucnc/how-to-publish-your-own-python-package-to-pypi-4318868210f9
I'm looking for a way to include some feature in a python (extension) module in installation phase.
In a practical manner:
I have a python library that has 2 implementations of the same function, one internal (slow) and one that depends from an external library (fast, in C).
I want that this library is optional and can be activated at compile/install time using a flag like:
python setup.py install # (it doesn't include the fast library)
python setup.py --enable-fast install
I have to use Distutils, however all solution are well accepted!
The docs for distutils include a section on extending the standard functionality. The relevant suggestion seems to be to subclass the relevant classes from the distutils.command.* modules (such as build_py or install) and tell setup to use your new versions (through the cmdclass argument, which is a dictionary mapping commands to classes which are to be used to execute them). See the source of any of the command classes (e.g. the install command) to get a good idea of what one has to do to add a new option.
An example of exactly what you want is the sqlalchemy's cextensions, which are there specifically for the same purpose - faster C implementation. In order to see how SA implemented it you need to look at 2 files:
1) setup.py. As you can see from the extract below, they handle the cases with setuptools and distutils:
try:
from setuptools import setup, Extension, Feature
except ImportError:
from distutils.core import setup, Extension
Feature = None
Later there is a check if Feature: and the extension is configured properly for each case using variable extra, which is later added to the setup() function.
2) base.py: here look at how BaseRowProxy is defined:
try:
from sqlalchemy.cresultproxy import BaseRowProxy
except ImportError:
class BaseRowProxy(object):
#....
So basically once C extensions are installed (using --with-cextensions flag during setup), the C implementation will be used. Otherwise, pure Python implementation of the class/function is used.
how to create a good plugin engine for standalone executables created with pyInstaller, py2exe or similar tools?
I do not have experience with py2exe, but pyInstaller uses an import hook to import packages from it's compressed repository. Of course I am able to import dynamically another compressed repository created with pyInstaller and execute the code - this may be a simple plugin engine.
Problems appears when the plugin (this what is imported dynamically) uses a library that is not present in original repository (never imported). This is because import hook is for the original application and searches for packages in original repository - not the one imported later (plugin package repository).
Is there an easy way to solve this problem? Maybe there exist such engine?
When compiling to exe, your going to have this issue.
The only option I can think of to allow users access with thier plugins to use any python library is to include all libraries in the exe package.
It's probably a good idea to limit supported libraries to a subset, and list it in your documentation. Up to you.
I've only used py2exe.
In py2exe you can specify libraries that were not found in the search in the setup.py file.
Here's a sample:
from distutils.core import setup
import py2exe
setup (name = "script2compile",
console=['script2compile.pyw'],
version = "1.4",
author = "me",
author_email="somemail#me.com",
url="myurl.com",
windows = [{
"script":"script2compile.pyw",
"icon_resources":[(1,"./ICONS/app.ico")] # Icon file to use for display
}],
# put packages/libraries to include in the "packages" list
options = {"py2exe":{"packages": [ "pickle",
"csv",
"Tkconstants",
"Tkinter",
"tkFileDialog",
"pyexpat",
"xml.dom.minidom",
"win32pdh",
"win32pdhutil",
"win32api",
"win32con",
"subprocess",
]}}
)
import win32pdh
import win32pdhutil
import win32api
PyInstaller does have a plugin system for handling hidden imports, and ships with several of those already in. See the webpage (http://www.pyinstaller.org) which says:
The main goal of PyInstaller is to be compatible with 3rd-party packages out-of-the-box. This means that, with PyInstaller, all the required tricks to make external packages work are already integrated within PyInstaller itself so that there is no user intervention required. You'll never be required to look for tricks in wikis and apply custom modification to your files or your setup scripts. Check our compatibility list of SupportedPackages.