We Keep Coding

Prevent Docker from installing python package requirements on every build (without requirements.txt)

I'm building an image from a Dockerfile where my main program is a python application that has a number of dependencies. The application is installed via setup.py and the dependencies are listed inside. There is no requirements.txt. I'm wondering if there is a way to avoid having to download and build all of the application dependencies, which rarely change, on every image build. I saw a number of solutions that use the requirements.txt file but I'd like to avoid having one if possible.

You can use requires.txt from the egg info to preinstall the requirements.
WORKDIR path/to/setup/script
RUN python setup.py egg_info
RUN pip install -r pkgname.egg-info/requires.txt

One solution is: if those dependencies rarely change as you said what you could do is another image with already those packages installed. You would create that image and then save it using docker save, so you end with a new base image with the required dependencies. docker save will create a .tar with the image. You have to load the image using docker load and then in your Dockerfile you would do:
FROM <new image with all the dependencies>
//your stuff and no need to run pip install
....
Hope it helps
Docker save description

Deploying a custom python package with `pip`

I have a custom Python package (call it MyProject) on my filesystem with a setup.py and a requirements.txt. This package needs to be used by a Flask server (which will be deployed on AWS/EC2/EB).
In my Flask project directory, I create a virtualenv and run pip install -e ../path/to/myProject.
But for some reason, MyProject's upstream git repo shows up in pip freeze:
...
llvmlite==0.19.0
-e git+https://github.com/USERNAME/MYPROJECT.git#{some-git-hash}
python-socketio==1.8.0
...
The reference to git is a problem, because the repository is private and the deployment server does not (and should not, and will never) have credentials to access it. The deployment server also doesn't even have git installed (and it seems extremely problematic that pip assumes without my permission that it does). There is nothing in MyProject's requirements.txt or setup.py that alludes to git, so I am not sure where the hell this is coming from.
I can dupe the project to a subdirectory of the Flask project, and then put the following in MyFlaskProject's requirements.txt:
...
llvmlite==0.19.0
./MyProject
python-socketio==1.8.0
...
But this doesn't work, because the path is taken as relative to the working directory of the pip process when it is run, not to requirements.txt. Indeed, it seems pip is broken in this respect. In my case, EC2 runs its install scripts from some other directory (with a full path to requirements.txt specified), and as expected, this fails.
What is the proper way to deploy a custom python package as a dependency of another project?

To install your own python package from a git repo you might want to check this post
To sort out the credential issue, why not having git installed on the EC2? You could simply create an ssh key and share it with MyProject repository.
I am using this solution on ECS instances deployed by Jenkins (with Habitus to hide Jenkin's ssh keys while building the image) and it works fine for me!

running python sdist on a perforce project fails

We are using perforce for our version control. So when i go to build a package
python setup.py sdist
it creates a versioned folder with everything in it and then tries to delete it and fails because it contains a bunch of read-only files. It also fails when trying to write to setup.cfg when that isn't checked out.
What is the proper way to do this from a perforce repo or any package that may have read-only files? Do i have to check out everything first and revert unchanged?
Thanks

Python "setup.py develop": is it possible to create ".egg-info" folder not in source code folder?

Python has ability to "pseudoinstall" a package by running it's setup.py script with develop instead of install. This modifies python environment so package can be imported from it's current location (it's not copied into site-package directory). This allows to develop packages that are used by other packages: source code is modified in place and changes are available to rest of python code via simple import.
All works fine except that setup.py develop command creates an .egg-info folder with metadata at same level as setup.py. Mixing source code and temporary files is not very good idea - this folder need to be added into "ignore" lists of multiple tools starting from vcs and ending backup systems.
Is it possible to use setup.py develop but create .egg-info directory in some other place, so original source code is not polluted by temporary directory and files?

setup.py develop creates a python egg, in-place; it does not [modify the] python environment so package can be imported from it's current location. You still have to either add it's location to the python search path or use the directory it is placed in as the current directory.
It is the job of the develop command to create an in-place egg, which may include compiling C extensions, running the 2to3 python conversion process to create Python3 compatible code, and to provide metadata other python code may be relying on. When you install the package as an egg in your site-packages directory, the same metadata is included there as well. The data is certainly not temporary (it is extracted from your setup.py file for easy parsing by other tools).
The intent is that you can then rely on that metadata when using your package in a wider system that relies on the metadata being present, while still developing the package. For example, in a buildout development deployment, we often use mr.developer to automate the process of fetching the source code for a given package when we need to work on it, which builds it as a develop egg and ties it into the deployment while we work on the code.
Note that the .egg-info directory serves a specific purpose: to signal to other tools in the setuptools eco-system that your package is installed and available. If your package is a dependency of another egg in your setup, then that dependency is satisfied. pip and easy_install and buildout will not try and fetch the egg from somewhere else instead.
Apart from creating the .egg-info directory, the only other thing the command does, is to build extensions, in-place. So the command you are looking for instead is:
setup.py build_ext --inplace
This will do the exact same thing as setup.py develop but leave out the .egg-info directory. It also won't generate the .pth file.
There is no way of generating only the .pth file and leave out the .egg-info directory generation.
Technically speaking, setup.py develop will also check if you have the setuptools site.py file installed to support namespaced packages, but that's not relevant here.

The good manner is to keep all source files inside special directory which name is your project name (programmers using other languages keep their code inside src directory). So if your setup.py file is inside myproject directory then you should keep the files at myproject/myproject. This method keeps your sources separated from other files regardless what happen in main directory.
My suggestion would be to use whitelist instead of blacklist -- tell the tools to ignore all files excluding these which are inside myproject directory. I think that this is the simplest way not to touch your ignore lists too often.

Try the --install-dir option. You may also want to use --build-dir to change building dir.

Delete files after upload to PyPi

I'm uploadig my package to PyPi with this command:
python setup.py sdist upload
This command generate some files and folders, is there any option to delete this files after upload?

The sdist command calls the build command which by default puts files in a build subdirectory. You probably want to keep that around (i.e. not care about it) to speed up future builds.
sdist then puts the distribution files in a dist subdirectory by default. python setup.py sdist -d $TMP (or the %something% equivalent environment variable for Windows) can be used to put the file in a temporary directory instead, so that they’re wiped out at the next boot.
If you really care about the build dir, try this: python setup.py build -b $TMP sdist -d $TMP. sdist should be clever enough to find the files created by build.
distutils docs: http://docs.python.org/distutils
command help: python setup.py build --help

Log into PyPI, and click on your package in the gray box in the upper right corner of the screen. Click "files" in the list to the right of the appropriate version. That will load a new page. Check off the files you want to delete and click the gray "Remove" button.