I have a python package that is used by other applications across an organization, let's call it buildtools.
Other applications in my organization have installed this package via
pip install git+https://${OAUTH_TOKEN}:x-oauth-basic#github.com/my_organization/buildtools#egg=buildtools
I want to add a new feature to buildtools that requires a 3rd party package, let's just say its requests. So within buildtools I add requests to requirements.txt, import it, and it's all good.
But none of the other applications in my organization have requests as one of their dependencies in requirements.txt.
When I merge my new code in and update the package, I believe we will run into some ImportError: No module named requests errors in the downstream applications that use buildtools.
How can I ensure that any application that uses the buildtools package gets the requests package installed when they get the latest buildtools?
In other words, how can I update buildtools's dependencies recursively?
I am aware that I could add requests to requirements.txt across all the applications in my organization that uses buildtools, but I'm trying to avoid that.
Why don't you just run
pip install -r requirements.txt
as discussed, e.g. here?
That's the best and most painless way to update/install required packages recursively.
After further research I found that install_requires within setup.py is exactly what I was looking for. This example explains it well.
Related
I have an internal utility library that is used by many projects. There is quite a bit of overlap between the projects in the code they pull from the utility library, but as the library grows, so does the amount of extra stuff any individual project gets that it won't be using. This wouldn't be an issue if the library consisted of only python, but the library also bundles in binaries.
Example-
psycopg2 is used in a handful of places within the utility library, but not all projects need db access. Because the development environment isn't the same as the production environment, the utility library also includes psycopg2 binaries for the prod environment.
This grows with openssl libraries, pandas, numpy, scipy, pyarrow, etc. The result is that a small, 50 line, single purpose script that may need db access is bundled into a 100mb+ deployment package.
So what I'd like to do is carve up the utility library into pieces where the projects down stream can choose which pieces to pull in, but keep the utility library code in one easy-to-manage place. That way, this small single purpose application can choose to import internal-util#core, internal-util#db and not include internal-util#numpy and internal-util#openssl
Is what i'm describing possible to do?
Not directly, to my best knowledge. pip installs a package fully, or not at all.
However, if you're careful in your package about how you import things that may require psycopg2 or someotherlargebinarything, you could use the extras_require feature and thus have the package's users choose which dependencies they want to pull in:
setup(
# ...
name='myawesometoolbelt',
extras_require={
'db': ['psycopg2'],
'math': ['numpy'],
},
)
and then, in your requirements.txt, or pip invocation,
myawesometoolbelt[db,math]
Have you tried looking at pip freeze > requirements.txt and pip install -r requirements.txt?
Once you generated your pip list via pip freeze, it is possible to edit which packages you want installed and which to omit from the requirements.txt generated.
You can then pip install -r requirements.txt the things you want back in.
Is there a way to include packages/modules not available through pip in the requirements file so that the project is portable?
The default version of lxml seems to have issues with pypy so I need to use a custom fork.
The problem is I need Heroku (where I deploy this application) to use a custom version of lxml and not the one that's available via pip. Is there any way to do this?
You can by using the listed git packages syntax, you would need to add the following line to your requirements.txt
-e git://github.com/aglyzov/lxml.git#egg=lxml
I am currently developing a command line application in python, which I will be uploading to pypi for end users to pip install and use. I am taking advantage of the extras functionality in setup.py to support 2 versions of my application, one is a basic functionality version with minimal dependencies and the other is more feature rich but has a large amount of dependencies (numpy, pandas, networkx, matplotlib, etc)
So briefly:
pip install app # simple, no deps
pip install app[all] # all the deps
Now the problem is that one of my dependencies in the feature rich version has what has been described as "flakey" pypi support. Basically, it cannot be installed unless one of its dependencies is already pre-installed before the whole installation process occurs. Luckily (or not), my application (which pulls in this flakey module) also has the flakey modules needed module. Lets refer to the module that flakey module needs pre-installed fixer-module
So:
pip install app[all] # triggers the installation of flakey module
Installation will fail here if fixer-module is not installed, even though it will be installed before flakey module is. One must basically do this:
pip install fixer-module
pip install app[all]
Now what I would like to do is include some kind of checking code that accomplishes the following:
Runs only when the app[all] distribution is being installed
Does a try import fixer-module, except ImportError check and prints a message explaining the situation.
Stops and cleans up the installation process before it fails
I have been researching this for quite some time. I found some examples of checking the input args to setup.py and whatnot but none of them seem to cover how to handle stuff during the end user install process. Any pointers are much appreciated!
I am trying to build a package for an apps in python. It uses sklearn, pandas, numpy, boto and some other scientific module from anaconda. Being very unexperienced with python packaging, I have various questions:
1- I have some confidential files .py in my project which I don't want anyone to be able to see. In java I would have defined private files and classes but I am completely lost in python. What is the "good practice" to deal with these private modules? Can anyone link me some tutorial?
2- What is the best way to package my apps? I don't want to publish anything on Pypi, I only need it to execute on Google App engine for instance. I tried a standalone package with PyInstaller but I could not finish it because of numpy and other scipy packages which makes it hard. Is there a simple way to package in a private way python projects made with anaconda?
3- Since I want to build more apps in a close future, shall I try to make sub-packages in order to use them for other apps?
The convention is to lead with a single underscore _ if something is internal. Note that this is a convention. If someone really wants to use it, they still can. Your code is not strictly confidential.
Take a look at http://python-packaging-user-guide.readthedocs.org/en/latest/. You don't need to publish to pypi to create a Python package that uses tools such as pip. You can create a project with a setup.py file and a requirements.txt file and then use pip to install your package from wherever you have it (e.g., a local directory or a repository on github). If you take this approach then pip will install all the dependencies you list.
If you want to reuse your package, just include it in requirements.txt and the install_requires parameter in setup.py (see http://python-packaging-user-guide.readthedocs.org/en/latest/requirements/). For example, if you install your package with pip install https://github/myname/mypackage.git then you could include https://github/myname/mypackage.git in your requirements.txt file in future projects.
My team uses git to track the code and we want to do the following: (1) One team-member installs a third-party python package and (2) makes it available on our git repo, so that the rest of the team can simply install the package by pulling the latest version of our code.
Is that possible at all? If so, what are feasible solutions? If not, what approach works best in your experience?
Background: We are using python 2.7.*. The third-party package is testfixtures to unittest the logging of our software. We use Windows and Mac.
Create a requirements.txt file and place it in your directory. Each line in the file should contain the name of a package that your entire team should have installed. Your team members, once they have the new version of the file, can run pip install -r requirements.txt. Then, update the requirements.txt file every time you have a new package required, and rerun the command.
Some editors (like PyCharm) will even automatically detect a requirements file and prompt you to install the packages.