It's a slightly awkwardly worded question, but basically:
I can only activate python packages by adding to $PYTHONPATH
I want to be able to use arbitrary python packages this way
Some python packages use package namespaces
I've got a wrapper script to build a python package, it mostly just does setup.py --single-version-externally-managed. I can add whatever I want to this build process - but it must be general, as it's used to build a large number of packages.
Normally, I can just add the build result onto $PYTHONPATH, and all is well.
But with namespaced packages (e.g ndg-httpsclient), it looks a lot like they'll only work when in a designated site-packages directory, because they use .pth files.
Confusingly there is a ndg/__init__.py in the source, which contains the namespace package boilerplate code:
__import__('pkg_resources').declare_namespace(__name__)
So that looks like it ought to be importable directly on $PYTHONPATH, but then the installation taunts me with:
Skipping installation of /<some-build-dir>/site-packages/ndg/__init__.py (namespace package)
Presumably it has a good reason for doing that, but its reliance on .pth files means the result can't be used via $PYTHONPATH, only by copying it to a well-known location, which I am unable to do for technical reasons. I am also unable to inject arbitrary python into the app initialization (so site.addsitedir() is out too).
So, is there some way I don't know about to have a namespace module importable without putting it in a well-known location? Alternatively, is there a way to convince setuptools to not skip the __init__.py -- and would that actually work?
Related
Running
$ python setup.py build_ext
with the usual Cython extension configuration creates a build directory and places the compiled modules deep within it.
How is the Python interpreter supposed to find them now? Should PYTHONPATH include those sub-sub-directories? It seems kludgy to me. Perhaps this is meant to work differently?
You will find the information here https://docs.python.org/3.5/install/
Build is an intermediary result before python actually install the module. Put in pythonpath the paths to the library that is, for instance:
<dir>/local/lib/python
if you use the "home" installation technique and dir is the directory you have chosen, ex
/home/user2
Presumably, when you write a package containing Cython code, your setup.py will contain something similar to this:
setup(
ext_modules = cythonize("example.pyx")
)
(there are some variations, but that's the general idea). When you run
python setup.py install
or
python setup.py install --user
You will see it creates binary files (with extensions based on your OS - on mine it will be example.so) and copies them to the standard installation directory (also depending on your OS).
These binary files are therefore already in the import path of your Python distribution, and it can import them like regular modules.
Consequently, you do not need to add the build directory to the path. Just install (possibly with --user, or use virtualenv, if you're developing), and let the extensions be imported the regular way.
Java has the concept of packaging all of the code into a file called a Jar file. Does Python have an equivalent idea? If so, what is it? How do I package the files?
Python doesn't have any exact equivalent to a .jar file.
There are many differences, and without knowing exactly what you want to do, it's hard to explain how to do it. But the Python Packaging User Guide does a pretty good job of explaining just about everything relevant.
Here are some of the major differences.
A .jar file is a compiled collection of classes that can be dropped into your application, or installed anywhere on your CLASSPATH.
In Python:
A .py (or .pyc) module can be dropped into your application, or installed anywhere on your sys.path, and it can be imported and used.
A directory full of modules can be treated the same way; it becomes a package (or, if it doesn't contain an __init__.py, it merges with other directories of the same name elsewhere on sys.path into a single package).
A .zip archive containing any number of modules and packages can be stored anywhere, and its path added to your sys.path (e.g., at runtime or via PYTHONPATH) and all of its contents become importable.
Most commonly, you want things to be installed into a system, user, or virtualenv site-packages directory. The recommended way to do that is to create a pip-compatible package distribution; people then install it (and possibly automatically download it from PyPI or a private repo) via pip.
pip does a lot more than that, however. It also allows you to manage dependencies between packages. So ideally, instead of listing a bunch of prereqs that someone has to go download and install manually, you just make them dependencies, and someone just has to pip install your-library. And it keeps track of the state of your site-packages, so you can uninstall or upgrade a package without having to track down the specific files.
Meanwhile, in Java, most .jar files are cross-platform; build once, run anywhere. A few packages have JNI native code and can't be used this way, but it's not the norm.
In Python, many packages have C extensions that have to be compiled for each platform, and even pure-Python packages often need to do some install-time configuration. And meanwhile, "compiling" pure Python doesn't do anything that can't be done just as well at runtime. So in Python, you generally distribute source packages, not compiled packages.
However, .wheel is a binary package format. You can pip wheel to build binary packages for different targets from the source package; then, if someone tries to pip install your package, if there's a wheel for his system, that will be downloaded and installed.
Easy Install from setup_tools defines the .egg format for deploying Python libraries or applications. While similar to JAR, it is nowhere spread as universally as JARs in Java world. Many people just deploy the .py files.
A newer format, intended to supersede eggs, is wheel.
Though it's not a perfect susbstitute of jar due to portability issues, I would add the "auto-extracting" archive way.
One possibility is "makeself": https://makeself.io/
But if you don't need to package external files, and if you like KISS approach, the following is a nice and clean alternative:
The following is taken from Asim Jalis's website.
How to deploy a Python application as a zip file
Create a file __main__.py containing:
print "Hello world from Python"
Zip up the Python files (in this case just this one file) into app.zip by typing:
zip app.zip *
The next step adds a shebang to the zip file and saves it as app—at this point the file app is a zip file containing all your Python sources.
echo '#!/usr/bin/env python' | cat - app.zip > app
chmod 755 app
That’s it. The file app is now have a zipped Python application that is ready to deploy as a single file.
You can run app either using a Python interpreter as:
python app
Or you can run it directly from the command line:
./app
Reference: https://gist.github.com/asimjalis/4237534
I have a simple script that has a dependency on dnspython for parsing zone files. I would like to distribute this script as a single .py that users can run just so long as they have 2.6/2.7 installed. I don't want to have the user install dependencies site-wide as there might be conflicts with existing packages/versions, nor do I want them to muck around with virtualenv. I was wondering if there was a way to embed a package like dnspython inside the script (gzip/base64) and have that script access that package at runtime. Perhaps unpack it into a dir in /tmp and add that to sys.path? I'm not concerned about startup overhead, I just want a single .py w/ all dependencies included that I can distribute.
Also, there would be no C dependencies to build, only pure python packages.
Edit: The script doesn't have to be a .py. Just so long as it is a single executable file.
You can package multiple Python files up into a .egg. Egg files are essentially just zip archives with well defined metadata - look at the setuptools documentation to see how to do this. Per the docs you can make egg files directly executable by specifying the entry point. This would give you a single executable file that can contain your code + any other dependencies.
EDIT: Nowadays I would recommend building a pex to do this. pex is basically an executable zip file with non stdlib dependencies. It doesn't contain a python distribution (like py2app/py2exe) but holds everything else and can be built with a single command line invocation. https://pex.readthedocs.org/en/latest/
The simplest way is just to put your python script named __main__.py with pure Python dependencies in a zip archive, example.
Otherwise PyInstaller could be used to produce a stand-alone executable.
please don't do this. If you do DO NOT make a habit of it.
pydns is BDS licensed but if you try to "embed" a gpl module in this way you could get in trouble
you can learn to use setuptools and you will be much happier in the long run
setuptools will handle the install of dependencies you identified (I'm not sure if the pydns you are using is pure python so you might create problems for your users if you try to add it yourself without knowing their environment)
you can set a url or pypi so that people could upgrade your script with easy_install -U
I have two packages that install different packages with the same name. They are both "packages" in that they have top-level setup.py files which specify package=['foo'] in the setup command.
If I install using distutils.core, the last to be installed overwrites the previous one (but I think wouldn't overwrite unless the .py files all had the same names?). If I install using setuptools, the two packages get installed into different eggs.
One option would be to explicitly set sys.path before importing the package name; this seems "un-pythonic" and rather dirty.
Assuming I have these two identically named packages installed in different eggs from setuptools, how do I specify which is imported?
Setuptools guide mentions --multi-version (-m) switch that removes package from sys.path completely. You have to use pkg_resources.require('package==version') in your code as early as possible to let it fix sys.path. This advice is what easy_install always prints when one uses -m.
But you can't have both imported at once (unless they're designed to do so using namespace packages).
I think the best way to workaround, would be to change the name of the toplevel directory, unless other packages depend on that package.
You can do this by altering the setup.py or just change the name folder in site-packages directly. The egg is just meta data.
As far as setting sys.path, it's better to use the site module, by creating a .pth file. When instantiated, any paths located in that file will be added to the "head" of the python path.
Are these two packages different and the naming is a coincidence, or are they simply branches of the same?
How can I specify an arbitrary set of files (non necessarily .py files) so that they are distributed and installed just like normal Python modules?
Some background
I am using distutils to distribute and install my Python library. One of the modules in this library imports a 3rd-party Python extension called bpy.so (this is a Blender module). The bpy.so extension requires some other files, as well. I want to distribute and install bpy.so and the additional required files with my library.
One way to do this is by specifying all of the extra files as data_files to setup(). However, I don't know how to reliably specify the same installation directory as is used for my pure python modules (e.g. /usr/local/lib/python3.2/dist-packages).
I can distribute the extra files by creating a MANIFEST.in file (see this question), but I also want to install the files.
I would suggest to use setuptools on top of distutils.
This is a good reference document to get started with it. The advantage of using setuptools is that it has a few nice features like the possibility to include all files under a given directory (package) automatically or according to filters. This is the section of the above document dealing with this specific aspect.
HTH!
PS: distutils' current situation/limitations are the source of much complain from most of its users. It seems a new generation of the library is on its way though!