Im currently refactoring my monolith into microserivces. To make them communicate each service has as client module with client in it that call per request the other services. I want to manage the different packages as easy as possible so I created a repository which is my package. Then each folder/module is the service with the modules of it that are needed.
What I want to achieve is that I can simply call pip install package["subpackage"] and it installs only that specific module of the package. I choose a big package over small packages because of the naming problem that most services have basic names where pip packages already exist with these names.
Repository of package
repo
payments/
client/
models/
auth/
client/
models/
setup.py
Is there a way to provide the information what each submodule / module needs for installing like install_requires for each module ?
Is there a other good approach that I should take ? I know some companies to it with java so that each module is its own "package" but they are all under a company package. Maybe in python there is a better solution for this
You can add additional requirements using:
setup(
name="package",
version=__version__,
install_requires=["stuff", "every", "package", "needs"],
extras_require={
"subpackage": ["dependency_1", "dependency_2"]
}
)
You can then run
pip install package[subpackage]
To install the extra dependencies.
In my organization, we
Split each server into their own repositories
Have a 'mono repo' that has all of the servers as git submodules
Make each server pip-installable with an internal pip server
Have a deployment repo that pins the versions of the servers we are using, as well as all the deployment scripts such as pushing docker contianers to local testing environments and terraform scripts to deploy to AWS.
Related
Let's say I am building an event-driven architecture, with Python microservices. I have a stream package handling all interactions with the streaming platform. It is supposed to be used internally only, in the company.
stream
__init__.py
produce.py
process.py
security.py
What are my options for sharing this package between all my Python microservices?
Should I publish my stream package so my projects can install it?
Is there some kind of Gradle for python including the Multi project feature?
You can package your Python code to reusable packages.
Poetry is a popular modern tool to manage your Python package.
Poetry, and other Python package managers like pip, can directly install private packages from an internal file server or Git link.
You don't have to publish your Python modules to install them with pip. See Packaging Python projects from the Python documentation to understand how to create installable packages (Wheel, .whl) from your project and stop once you've got the .whl file (don't have to upload to the public package index). Then, you can put them either into your company-internal Python package index (e.g., artifactory) or into an object storage service or Git LFS, from where your other microservices can access and install the package during build with
pip install /path/to/your/stream-0.0.1-py3-none-any.whl
For a package I've authored, I've done python setup.py sdist bdist_wheel, which generates some package artifacts in the dist/ directory. Now I'd like to run the package's unit tests in those artifacts. What's a good way to do it?
To be clear: an alternative would be to run the tests directly from the local source files, but I want to avoid that to make sure I'm testing the exact pre-built artifact users would be installing (as suggested here).
I'm using Python 3, and I'm on a Linux or Mac OS environment. My context is a build server that builds, tests, and then publishes artifacts (to a private PyPI-like repo) as commits are made to a Git repository.
If there's some other approach I should be using instead, I'm all ears.
What you can do is:
Create a virtual environment
Install your package
Run the tests against your installed library using tools like pytest, you can read more about pytest good practices here: http://pytest.org/dev/goodpractises.html
As pointed in the pytest docs take a look to tox as well for your CI server: http://pytest.org/dev/goodpractises.html#use-tox-and-continuous-integration-servers
This is a related question regarding how to test using the installed package: Force py.test to use installed version of module
I am currently about to deploy my Python application. I used to think that the only good way to install the pure Python app is to just copy the source code files along with the requirements file, and install packages listed in the requirements file (also, Python onbuild Docker image does suppose this way).
But I can see that folks often install their apps using setuptools like ./setup.py install(it seems that Warehouse project does this, for example).
Which of the two is considered a better practice?
What are the benefits from installing your app as a package?
I am trying to build a package for an apps in python. It uses sklearn, pandas, numpy, boto and some other scientific module from anaconda. Being very unexperienced with python packaging, I have various questions:
1- I have some confidential files .py in my project which I don't want anyone to be able to see. In java I would have defined private files and classes but I am completely lost in python. What is the "good practice" to deal with these private modules? Can anyone link me some tutorial?
2- What is the best way to package my apps? I don't want to publish anything on Pypi, I only need it to execute on Google App engine for instance. I tried a standalone package with PyInstaller but I could not finish it because of numpy and other scipy packages which makes it hard. Is there a simple way to package in a private way python projects made with anaconda?
3- Since I want to build more apps in a close future, shall I try to make sub-packages in order to use them for other apps?
The convention is to lead with a single underscore _ if something is internal. Note that this is a convention. If someone really wants to use it, they still can. Your code is not strictly confidential.
Take a look at http://python-packaging-user-guide.readthedocs.org/en/latest/. You don't need to publish to pypi to create a Python package that uses tools such as pip. You can create a project with a setup.py file and a requirements.txt file and then use pip to install your package from wherever you have it (e.g., a local directory or a repository on github). If you take this approach then pip will install all the dependencies you list.
If you want to reuse your package, just include it in requirements.txt and the install_requires parameter in setup.py (see http://python-packaging-user-guide.readthedocs.org/en/latest/requirements/). For example, if you install your package with pip install https://github/myname/mypackage.git then you could include https://github/myname/mypackage.git in your requirements.txt file in future projects.
I am writing a CLI python application that has dependencies on a few libraries (Paramiko etc.).
If I download their source and just place them under my main application source, I can import them and everything works just fine.
Why would I ever need to run their setup.py installers or deal with python package managers?
I understand that when deploying server side applications it is OK for an admin to run easy_install/pip commands etc to install the prerequsites, but for a script like CLI apps that have to be distributed as a self-contained apps that only depend on a python binary, what is the recommented approach?
Several reasons:
Not all packages are pure-python packages. It's easy to include C-extensions in your package and have setup.py automate the compilation process.
Automated dependency management; dependencies are declared and installed for you by the installer tools (pip, easy_install, zc.buildout). Dependencies can be declared dynamically too (try to import json, if that fails, declare a dependency on simplejson, etc.).
Custom resource installation setups. The installation process is highly configurable and dynamic. The same goes for dependency detection; the cx_Oracle has to jump through quite some hoops to make installation straightforward with all the various platforms and quirks of the Oracle library distribution options it needs to support, for example.
Why would you still want to do this for CLI scripts? That depends on how crucial the CLI is to you; will you be maintaining this over the coming years? Then I'd still use a setup.py, because it documents what the dependencies are, including minimal version needs. You can add tests (python setup.py test), and deploy to new locations or upgrade dependencies with ease.