How can I 'embed' a Python library in my own Python package?
Take the Requests library, for instance.
How could I integrate it into my own package, the objective being to allow me to run my application on different machines without actually installing Requests on every one, but having it in the same folder as my package?
Is this even possible?
If it's a pure python library (no compiled modules) you can simply place the library in a folder in your project and add that folder to your module search path. Here's an example project:
|- application.py
|- lib
| `- ...
|- docs
| `- ...
`- vendor
|- requests
| |- __init__.py
| `- ...
`- other libraries...
The vendor folder in this example contains all third party modules. The file application.py would contain this:
import os
import sys
# Add vendor directory to module search path
parent_dir = os.path.abspath(os.path.dirname(__file__))
vendor_dir = os.path.join(parent_dir, 'vendor')
sys.path.append(vendor_dir)
# Now you can import any library located in the "vendor" folder!
import requests
Bonus fact
As noted by seeafish in the comments, you can install packages directly into the vendor directory:
pip install <pkg_name> -t /path/to/vendor_dir
If you only need to run your application may be pyinstaller packaging is a better option.
It will create a single bundle with everything that is needed, including Python, to avoid dependencies on the system you're running in.
While not a direct answer to your question. You may want to look at setuptools. By leveraging this package distribution mechanism you can describe your dependencies and when your package is "installed" all the dependent packages will be installed too. You would create a setup.py file at the top of your package structure similar to:
from setuptools import setup, find_packages
setup(
name = 'MyPackage',
version = '1.0',
packages = find_packages(),
...
install_requires = ['requests'],
...
)
this would be installed by the user
python setup.py install
Requests would be automatically installed too.
All of the above answers are correct but the best solution is creating a standard package.
You can refer to this link:
https://packaging.python.org/tutorials/packaging-projects/
Related
CloudFunctions does not support private dependencies.
Following their recommendation (https://cloud.google.com/functions/docs/writing/specifying-dependencies-python#using_private_dependencies) I downloaded the package via:
pip install -t vendor foo
touch vendor/__init__.py
Which results in a directory:
vendor
|- foo-0.0.1.dist-info
|- INSTALLER
|- METADTA
|- RECORD
|- REQUESTED
|- top_level.txt
|- WHEEL
|- __init__.py
Now trying to import vendor.foo results in an error:
ModuleNotFoundError: No module named 'vendor.foo
Is there an import subtlety I am missing or how is this supposed to work?
There are multiple ways to do this.
relative imports, see here, also the guide you linked uses relative imports. If you import as import vendor.foo it will only work if you run the python script in the folder, where also the 'vendor' folder is.
adding the package location to PYTHONPATH environment variable. export PYTHONPATH = $PYTHONPATH":/path/to/vendor. This can also be done inside a python script using sys.path. I'm not sure how this is done in case of Gcloud
installing the package to a virtual environment. As shown here you can setup a virtual environment. One you activated this you should be able to pip install packages to it.
First approach should definatly work for you.
The problem was in foo. Forgot a toplevel __init__.py, package autodetection failed and it added no sources to the wheel.
Thus when downloading later, it created no package directory.
I am trying to generate Python bindings for a C++ shared library with SWIG and distribute the project with conda. The build process seems to work as I can execute
import mymodule as m
ret = m.myfunctioninmymodule()
in my build directory. Now I want to install files that are created (namely, mymodule.py and _mymodule.pyd) in my conda environment on Windows so that I can access them from everywhere. But where do I have to place the files?
What I have tried so far is to put both files in a package together with a __init__.py (which is empty, however) and write a setup.py as suggested here. The package has the form
- mypackage
|- __init__.py
|- mymodule.py
|- _mymodule.pyd
and is installed under C:\mypathtoconda\conda\envs\myenvironmentname\Lib\site-packages\mypackage-1.0-py3.6-win-amd64.egg. However, the python import (see above) fails with
ImportError: cannot import name '_mymodule'
It should be noted that under Linux this approach with the package works perfectly fine.
Edit: The __init__.py is empty because this is sufficient to build a package. I am not sure, however, what belongs in there. Do I have to give a path to certain components?
My team has a folder of several small projects in python3. Amongst them, we have a utility folder with several utility functions, that are used throughout the projects. But the way to import it is very uncomfortable. This is the structure we use:
temp_projects
util
storage.py
geometry.py
project1
project1.py
project2
project2.py
The problem is that the import in the projects looks terrible:
sys.path.insert(1, os.path.join(sys.path[0], '..'))
import util.geometry
util.geometry.rotate_coordinates(....)
Also, pycharm and other tools are having trouble understanding it and to supply completion.
Is there some neater way to do that?
Edit:
All of the projects and utils are very much work-in-progress and are modified often, so I'm looking for something as flexible and comfortable as possible
PYTHONPATH environment variable might be a way to go. Just set it to projects folder location:
PYTHONPATH=/somepath/temp_projects
and you'll be able to use util as follows:
import util.geometry
util.geometry.rotate_coordinates(....)
Also this will be recognized by PyCharm automatically.
I believe the correct route would be completely different than what you are doing right now. Each project should be stored in a different Git repository, share modules should be added as git submodules. Once those projects will get larger and more complex (and they probably will), it would be easier to manage them separately.
In short
Projects structure should be:
Project_1
|- utils <submodule>
|- storage.py
|- geometry.py
|- main.py
Project_2
|- utils <submodule>
|- storage.py
|- geometry.py
|- main.py
Working with submodules
### Adding a submodule to an existing git directory
git submodule add <git#github ...> <optional path/to/submodule>
### Pulling latest version from master branch for all submodules
git submodule update --recursive --remote
### Removing submodule from project
# Remove the submodule entry from .git/config
git submodule deinit -f path/to/submodule
# Remove the submodule directory from the project's .git/modules directory
rm -rf .git/modules/path/to/submodule
# Remove the entry in .gitmodules and remove the submodule directory located at path/to/submodule
git rm -f path/to/submodule
Further reading https://git-scm.com/book/en/v2/Git-Tools-Submodules
According to Importing files from different folder adding a __init__.py in the util folder will cause python to treat it as a package. Another thing you can do is use import util.geometry as geometry and then you could use geometry.rotate_coordinates(...) which also improves readability.
If you create a setup.py file for your util module you can install it simply with pip. It will handle everything for you. After installation you can import it across the system.
import util
pip install
# setup.py is in current folder
sudo -H pip3 install .
or if the util module itself is still under development you can install it with the -e editable option. Then it automatically updates the installation when you make changes to the code.
sudo -H pip3 install -e .
For the project administration I recommend to use git as #Michael liv. is suggesting, especially if you work in a team.
Use importlib.
import importlib, importlib.util
def module_from_file(module_name, file_path):
spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
geometry = module_from_file("geometry", "../temp_projects/utils/geometry.py")
geometry.rotate_coordinates(...)
Other option (I used this appoach in my project):
Lets assume that projects are ran from project1.py and project2.py files accordingly.
At the top of these files you can add following imports and actions:
import sys
import os
sys.path.append(os.path.join(os.getcwd(), os.pardir))
import your_other_modules
your_other_modules.py will contains following for impotring utils
from utils import storage
from utils import geometry
# or from project2 import project2, etc..
May be it's not a best way, but just like one more option. I hope it will be helpful for someone.
The source for the package is here
I'm installing the package from the index via:
easy_install hackertray
pip install hackertray
easy_install installs images/hacker-tray.png to the following folder:
/usr/local/lib/python2.7/dist-packages/hackertray-1.8-py2.7.egg/images/
While, pip installs it to:
/usr/local/images/
My setup.py is as follows:
from setuptools import setup
setup(name='hackertray',
version='1.8',
description='Hacker News app that sits in your System Tray',
packages=['hackertray'],
data_files=[('images', ['images/hacker-tray.png'])])
My MANIFEST file is:
include images/hacker-tray.png
Don't use data_files with relative paths. Actually, don't use data_files at all, unless you make sure the target paths are absolute ones properly generated in a cross-platform way insted of hard coded values.
Use package_data instead:
setup(
# (...)
package_data={
"hackertray.data": [
"hacker-tray.png",
],
},
)
where hackertray.data is a proper python package (i.e. is a directory that contains a file named __init__.py) and hacker-tray.png is right next to __init__.py.
Here's how it should look:
.
|-- hackertray
| |-- __init__.py
| `-- data
| |-- __init__.py
| `-- hacker-tray.png
`-- setup.py
You can get the full path to the image file using:
from pkg_resources import resource_filename
print os.path.abspath(resource_filename('hackertray.data', 'hacker-tray.png'))
I hope that helps.
PS: Python<2.7 seems to have a bug regarding packaging of the files listed in package_data. Always make sure to have a manifest file if you're using something older than Python 2.7 for packaging. See here for more info: https://groups.google.com/d/msg/python-virtualenv/v5KJ78LP9Mo/OiBqMcYVFYAJ
I'm developing an Python egg that has several .txt dependencies (they're templates used to generate files by the egg itself), and I'm struggling to get those dependencies copied to site-packages during setup.py install. According to the distribute documentation...
Filesystem of my package:
setup.py
package
|--- __init__.py
|--- main.py
|--- binary (calls main.py with pkg_resources.load_entry_point)
|--- templates
|--file1.txt
|--file2.txt
In setup.py:
setup(
[...]
eager_resources = ['templates/file1.txt', 'templates/file2.txt']
)
Within my package:
from pkg_resources import resource_string
tpl = resource_string(__name__, 'templates/file1.txt')
...this combination of configuration and filesystem should result in file1.txt and file2.txt being available through pkg_resources.resource_string. Unfortunately, they're not being copied to site-packages during setup.py install. What am I missing?
Thanks!
The information can be found in the setuptools documentation for including package data: https://setuptools.readthedocs.io/en/latest/setuptools.html#including-data-files
Basically, you just need to set include_package_data=True in your setup.py file. If you are using subversion or CVS, all versioned files will be included. If not, you can specify which files to include with a MANIFEST.in file.
I believe distribute supports this as well.
You can then access the files as you would without them being packaged. i.e. in main.py you could have:
import os.path
f = open(os.path.join(os.path.dirname(__file__),'templates','file1.txt'))
print f.read()
f.close()
and this would work in the packaged version as well. One caveat is that you will have to also set zip_safe = False in setup.py so that all the files are unzipped during installation.