I have a repository which contains protocol buffer messages and services, originally written for a Golang project. I need to write a Python microservice but I'm not 100% clued up on how Python dependency management works. I need to import the definitions into my new Python project and compile them to Python.
How should I import my protocol buffer definitions repository into my Python project and compile the protobufs for Python? I can't copy the .pb files into my new project as they're shared amongst a number of other projects.
Protobufs is installable via pip
$ pip install protobuf
Read Installing Python Modules in the documentation if you need help on using pip.
Related
Let's say I am building an event-driven architecture, with Python microservices. I have a stream package handling all interactions with the streaming platform. It is supposed to be used internally only, in the company.
stream
__init__.py
produce.py
process.py
security.py
What are my options for sharing this package between all my Python microservices?
Should I publish my stream package so my projects can install it?
Is there some kind of Gradle for python including the Multi project feature?
You can package your Python code to reusable packages.
Poetry is a popular modern tool to manage your Python package.
Poetry, and other Python package managers like pip, can directly install private packages from an internal file server or Git link.
You don't have to publish your Python modules to install them with pip. See Packaging Python projects from the Python documentation to understand how to create installable packages (Wheel, .whl) from your project and stop once you've got the .whl file (don't have to upload to the public package index). Then, you can put them either into your company-internal Python package index (e.g., artifactory) or into an object storage service or Git LFS, from where your other microservices can access and install the package during build with
pip install /path/to/your/stream-0.0.1-py3-none-any.whl
I have a project that used virtualenv to package a python 2.7 project with 3 dependencies I found in a requirements.txt file: boto3, botocore and pypdftk.
Boto3 is an Amazon Web Services SDK for Python, and so is botocore (I believe.) Pypdftk is some external library used for transforming PDFs in python.
Now I am supposed to get this project compressed to a zip and uploaded to AWS Lambda, a service for server less computing.
Additionally, AWS Lambda only supports the standard python 3.9 library & runtime. So because my project has these external libraries and dependencies, it seems the past developer used a virtualenv to:
package a deprecated version of python 2.7
package the dependencies listed in the requirements.txt file
AWS Lambda has a feature called Layers, where you can upload zipped binaries to extend the standard core python3 library.
In summary:
I am failing to understand how to upload my compressed python3.9 project.
Do I upload these dependencies separately in the AWS Lambda Layer?
OR does compressing my file inside a virtualenv take care of the dependencies?
Much like a Docker Container? The virtualenv ships with the compiler/interpreter & dependencies?
Upgrade your Python code/dependencies to 3.9, following the "how-to" here: https://docs.python.org/3/howto/pyporting.html
While it's possible to deploy 2.7 code using a Docker image (which, at least for now , is still provided by AWS), that's not a long-term solution and you'll almost certainly put in more work to make it happen.
For your other questions
package a deprecated version of python 2.7
Virtual environments won't let you do this. There are tools such as pyenv that do, but they won't work for a Lambda because the version of Python that's used to run your Lambda is part of the Lambda configuration, and cannot be replaced.
package the dependencies listed in the requirements.txt file
Yes, this is what a virtual environment can be used for. When you activate the virtual environment and run pip install, it will install the packages into the lib directory in the virtual environment.
To produce a Lambda deployment bundle you must ZIP your source code together with the installed packages. Making this a little more challenging, the packages are actually installed in lib/python3.9/site-packages/, and Lambda doesn't want to see that path; change into the directory while building the ZIP.
Here's is the official doc for producing Python bundles: https://docs.aws.amazon.com/lambda/latest/dg/python-package.html
Do I upload these dependencies separately in the AWS Lambda Layer?
You can, but that's only useful if you're going to share the dependencies between Lambdas. Otherwise it increases the complexity of your deployment with little benefit.
I have a C++ project for which I am developing a Python interface. For now I am using pybind11 since it seems pretty neat, and has some nice tools for building the extension module with CMake, which is how the main C++ project is built.
Via CMake I managed to get a shared library containing the interface functions to build, however now that I have it I don't know how to tell Python that it exists and make it import-able. I don't want to reconfigure the whole build of the project to be launched via Python (i.e. as described here with setuptools) because it is a big project and I am just providing a Python interface to part of it. So it would be preferable if I could just build the shared library for Python along with the rest of the C++ code, and then just later on run "setup.py install" to do whatever else needs to be done to make the shared library visible to Python.
Is this possible? Or do I need to do some other sort of refactoring, like make the main project build some other pure C++ libraries, which I then just link into the Python extension module library which is built separately via setuptools?
If you need to install just one binary module you can create an simple installer just for that module. Let's assume that you have a binary module foo.so (or foo.pyd if you are working on Windows) that is already built with your cmake-generated build script. Then you can create a simple setup setup script:
from setuptools import setup
setup(
name='foo',
version='0.1.2.3',
py_modules=['foo']
)
Then you need to add MANIFEST.in file to pick your binary module file:
include foo.so
So you need 3 files:
foo.so
MANIFEST.in
setup.py
Now you can do python setup.py install from your Python virtual environment, and your binary module will be installed in it. If you want to distribute your module, then it's better to install Python wheel package and create a .whl file: python setup.py bdist_wheel. Such "wheel" can later be installed with pip command. Note that binary modules must be installed on the same platform and Python version that was used to build those modules.
In python we do pip install to install some external libraries, modules etc.
If i have to create a reusable python module that can be used by several different APIs in an enterprise, is there a known way to create this module and ship it so that consuming applications just install it somehow and import the module rather that taking the source code from a common repository, creating a local module out of it and then do a module import.
Can someone educate me with what best practices we have in python for this use case ?
The flow is quite simple:
You create your lib.
You define setup.py file (versioning here is important).
You build your lib.
You upload it to a pypi server (either public or your private).
Other applications simply bump version and pip install from either public or your private pypi server (there is a flag in pip command tool to switch to other server).
You should start with learning either distutils or setuptools (my favourite) for points 2 and 3.
For a private pypi server you have to set it up. I've actually never done it by myself but I assume it can't be hard. Google pypi server.
I developed my first webserver app in Python.
It's a but unusual, because it does not only depend on python modules (like tornado) but also on some proprietary C++ libs wrapped using SWIG.
And now it's time to deliver it (to Linux platform).
Due to dependency on C++ lib, just sending sources with requirements.txt does not seem enough. The only workaround would be to have exact Linux installation to ensure binary compatibility of the lib. But in this case there will be problems with LD_PATH etc.
Another option is to write setup.py to create sdist and then deploy it with pip install.
Unfortunately that would mean I have to kill all instances of the server before installing my package. The workaround would be to use virtualenv for each instance though.
But maybe I'm missing something much simpler?
If you need the package to be installed by some user the easiest way will be to write the setup.py - but no just with simple setup function like most of installers. If you look at some packages, they have very complicated setup.py scripts which builds many things and C extensions with installation scripts for many external dependences.
The LD_PATH problem you can solve like this. If your application have an entry-point like some script which you save in python's bin directory (or system /usr/bin) you override LD_PATH like export LD_PATH="/my/path:$LD_PATH".
If your package is system service, like some servers or daemons, you can write system package, for example debian package or rpm. Debian has a lot of scripts and mechanism to point out the dependencies with packages.
So, if you need some system libraries on the list you write it down in package source and debian will install them when you will be installing your package. For example your package have dependencies for SWIG and other DEV modules, and your C extension will be built properly.