We Keep Coding

Mitmproxy and installation with yocto recipes

I want to install the third-party tool called Mitmproxy in an embedded Linux environment with a low footprint, e.g. a Raspberry PI.
Rather than install using a standard Raspbian OS, I want to customize the install with Yocto and define the packages. This involves writing recipes to customize the packages that are installed. A good example at a high leve is described here.
Unfortunately this involves appending various Python3 packages as well pip-mitmproxy as well as all the dependencies of pip-mitmproxy in the local.conf file, e.g.
IMAGE_INSTALL_append = " python3"
IMAGE_INSTALL_append = " pip-mitmproxy"
Different recipe files need to be added in this case.
The challenge that I've found is getting the right versions of all the dependencies. I looked import pkg_resources but it doesn't seem to give me everything. Is there another recommended approach perhaps by looking at the source code.

I created Python venv on normal Linux Mint so pip freeze shows me empty list
After installing mitmproxy I run again pip freeze and it shows me
blinker==1.4
Brotli==1.0.7
certifi==2020.6.20
cffi==1.14.1
click==7.1.2
cryptography==2.9.2
Flask==1.1.2
h11==0.9.0
h2==3.2.0
hpack==3.0.0
hyperframe==5.2.0
itsdangerous==1.1.0
Jinja2==2.11.2
kaitaistruct==0.8
ldap3==2.7
MarkupSafe==1.1.1
mitmproxy==5.2
passlib==1.7.2
protobuf==3.11.3
publicsuffix2==2.20191221
pyasn1==0.4.8
pycparser==2.20
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyperclip==1.8.0
ruamel.yaml==0.16.10
ruamel.yaml.clib==0.2.0
six==1.15.0
sortedcontainers==2.1.0
tornado==6.0.4
urwid==2.0.1
Werkzeug==1.0.1
wsproto==0.15.0
zstandard==0.13.0
But it is hard to say if it needs some C/C++ libraries for this.
For example pyOpenSSL will need library OpenSSL but I can't find it in setup.py or setup.cfg. There is only information about openssl-devel which means it may need openssl.
It seems the only way is to install and run it to see error messages :)

how to download all the python packages mentioned in the requirement.txt to a folder in linux?

I want to download all the python packages mentioned in the requirement.txt to a folder in Linux. I don't want to install them. I just need to download them.
python version is 3.6
list of packages in the requirement.txt
aiodns==0.3.2
aiohttp==1.1.5
amqp==1.4.7
anyjson==0.3.3
astroid==1.3.2
asyncio==3.4.3
asyncio-redis==0.14.1
billiard==3.3.0.20
blist==1.3.6
boto==2.38.0
celery==3.1
pexpect==4.0
pycryptodomex==3.7.0
pycurl==7.19.5.1
pyinotify==0.9.6
pylint==1.4.0
pyminifier==2.1
pyOpenSSL==0.15.1
pypacker==2.9
pyquery==1.2.9
pysmi==0.3.2
pysnmp==4.4.4
PyStaticConfiguration==0.9.0
python-daemon==2.1.2
python-dateutil==2.4.2
python-ldap==3.2.0
python-libnmap==0.6.2
python-otrs==0.4.3
pytz==2015.4
PyYAML==3.11
query-string==0.0.2
queuelib==1.2.2
redis==2.10.3
requests==2.22.1
requests-aws4auth==0.9
requests-oauthlib==0.5.0
requests-toolbelt==0.5.0
scp==0.10.2
six==1.10.0
South==1.0.1
tlslite==0.4.9
u-msgpack-python==2.1
urllib3==1.14
w3lib==1.12.0
websockets==3.3
Werkzeug==0.10.4
xlrd==1.0.0
XlsxWriter==1.0.5
zope.interface==4.1.2
GitPython==2.1.3

The documentation gives what you want : pip download
pip download does the same resolution and downloading as pip install,
but instead of installing the dependencies, it collects the downloaded
distributions into the directory provided
source
So you may try these option with pip download :
pip download -r requirement.txt -d your_directory

Python pip install cannot find module

So I am trying to install pylint using pip, as my work machine is offline I have downloaded pylint using pip and transferred this using a CD. As part of pylint download it also brought down asteroid, colorama, isort, lazy_object_proxy, McCabe, six, typed ast and wrapt.
However when running the install for pylint using the following command inside the directory with all the above files in:
python -m pip install --no-index --find-links . -r requirements.txt
This starts to work with it collecting pylint, isort and a couple of others, however after collecting asteroid it goes to collect lazy object proxy (which is in the directory) and gives the following error:
Could not find a version that satisfies the requirement lazy-object-proxy (from asteroid<3,>=2.2.0->pylint->-r requirements.txt (line 1)) (from versions: ) No matching distribution found for lazy-object-proxy (from asteroid<3,>=2.2.0->pylint->-r requirements.txt (line 1))
The version of lazy object proxy downloaded is 1.4.1
Im fairly new to this so maybe there is something in this error that highlights why it doesn't see or like the version that is downloaded and in the directory, any help would be much appreciated.
OS is windows 7, running python version 3.6.0
NOTE: even trying to just install lazy object proxy on its own fails saying it doesn't exist, its like its not there although it is in the folder.

Installing Python packages from local file system folder with pip

I am trying to install python packages from the local file system with pip as discussed in this question.
I have used pip2pi as suggested in the accepted answer in the above question.
dmanna#ubuntu:~$ mkdir -p pyt/pkg
dmanna#ubuntu:~$ pip2tgz pyt/pkg/ patroni[zookeeper]
The downloaded packages
dmanna#ubuntu:~$ ls /home/dmanna/pyt/pkg/
cdiff-1.0.tar.gz
certifi-2018.4.16-py2.py3-none-any.whl
chardet-3.0.4-py2.py3-none-any.whl
click-6.7-py2.py3-none-any.whl
idna-2.7-py2.py3-none-any.whl
kazoo-2.5.0-py2.py3-none-any.whl
patroni-1.4.4.tar.gz
prettytable-0.7.2.tar.bz2
psutil-5.4.6.tar.gz
psycopg2-2.7.4.tar.gz
python_dateutil-2.7.3-py2.py3-none-any.whl
pytz-2018.4-py2.py3-none-any.whl
PyYAML-3.12.tar.gz
requests-2.19.1-py2.py3-none-any.whl
six-1.11.0-py2.py3-none-any.whl
tzlocal-1.5.1.tar.gz
urllib3-1.23-py2.py3-none-any.whl
Then when I am trying to install the required package from the above local directory. It is giving me the below error
dmanna#ubuntu:~$ sudo pip install --no-index --find-links=pyt/pkg/patroni-1.4.4.tar.gz patroni[zookeeper]
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking patroni[zookeeper]
Running setup.py (path:/tmp/pip_build_root/patroni/setup.py) egg_info for package patroni
Installing extra requirements: 'zookeeper'
Downloading/unpacking urllib3>=1.19.1,!=1.21 (from patroni[zookeeper])
Could not find any downloads that satisfy the requirement urllib3>=1.19.1,!=1.21 (from patroni[zookeeper])
Cleaning up...
No distributions at all found for urllib3>=1.19.1,!=1.21 (from patroni[zookeeper])
Storing debug log for failure in /home/dmanna/.pip/pip.log
Can someone let me know what I am doing wrong?
Python - 2.7.6
PIP - 1.5.4
Ubuntu 14.04

--find-links=pyt/pkg
Find links in the directory, not in the patroni-1.4.4.tar.gz archive.

Pip ignores dependency_links in setup.py despite proper format

Looking as solutions in the past such as
pip ignores dependency_links in setup.py, this configuration should work.
Relevant content of my setup.py
packages=find_packages(),
dependency_links=['http://github.com/koji-project/koji/tarball/master#egg=koji'],
install_requires=['jira', 'PyYAML', 'requests', 'psycopg2',
'elasticsearch', 'beanbag', 'pyzabbix', 'enum34',
'beautifulsoup4', 'pytz', 'koji'],
tests_require=['flake8', 'autopep8', 'mock'],
include_package_data=True,
cmdclass={'test': setupTestRequirements}
The only thing I can think of is that my url is invalid. I don't see why it would be since it is of version 1.14.0.
Upon running pip install . I get.
Could not find a version that satisfies the requirement koji (from MARs==0.17.10) (from versions: ) No matching distribution found for koji (from MARs==0.17.10)
Upon running python setup.py develop --user, the output doesn't mention Koji

Your configuration is correct. However the problem lies elsewhere. Take a look at the koji repo on github: the project has no setup.py committed. As long as there's no setup.py script, neither pip nor setuptools (via setup.py install/setup.py develop) won't be able to install your project because they won't be able to install koji dependency as it is no valid python package at all.
Update:
The problem with koji repo on github is that it is only a mirror of the actual dev repo located on Fedora Pagure and is not synced with the upstream. So the correct answer is to use the real development repository instead of the github mirror:
dependency_links=['git+https://pagure.io/koji.git#egg=koji-1.14.0']
Easy peasy. :-)
Original answer (obsolete, only if you want to install from kojis repo mirror on Github):
I see two ways out of this situation:
Forking
fork koji on github
write your own setup.py script or copy it somewhere (see below for more info), commit and push
adapt the URL in dependency_links in your project's setup.py.
For testing, I prepared a fork of koji with a setup script; if I use its URL instead of the upstream repo, the installation succeeds. I also tagged my own "release" with koji-1.14.0.post1 to distinct the version with the setup script from the vanilla ones. Example setup.py with the new dependency:
from setuptools import setup, find_packages
setup(
name='spam',
version='0.1',
author='nobody',
author_email='nobody#nowhere.com',
url='www.example.com',
packages=[],
dependency_links=['https://github.com/hoefling/koji/tarball/master#egg=koji-1.14.0.post1'],
install_requires=['koji==1.14.0.post1'],
)
Testing the installation with pip yields:
$ pip install . --process-dependency-links
Obtaining file:///home/hoefling/python/spam
DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release.
Collecting koji==1.14.0.post1 (from spam==0.1)
Downloading https://github.com/hoefling/koji/tarball/master (1.4MB)
100% |████████████████████████████████| 1.4MB 759kB/s
Collecting pyOpenSSL (from koji==1.14.0.post1->spam==0.1)
Using cached pyOpenSSL-17.5.0-py2.py3-none-any.whl
Collecting pycurl (from koji==1.14.0.post1->spam==0.1)
Using cached pycurl-7.43.0.1.tar.gz
...
Installing collected packages: six, idna, asn1crypto, pycparser, cffi,
cryptography, pyOpenSSL, pycurl, python-dateutil, chardet, certifi,
urllib3, requests, pykerberos, requests-kerberos, rpm-py-installer,
koji, spam
Running setup.py install for rpm-py-installer ... done
Running setup.py install for koji ... done
Running setup.py install for spam ... done
Successfully installed asn1crypto-0.23.0 certifi-2017.11.5 cffi-1.11.2
chardet-3.0.4 cryptography-2.1.4 idna-2.6 koji-1.14.0.post1 pyOpenSSL-17.5.0
pycparser-2.18 pycurl-7.43.0.1 pykerberos-1.1.14 python-dateutil-2.6.1
requests-2.18.4 requests-kerberos-0.11.0 rpm-py-installer-0.5.0 six-1.11.0
spam-0.1 urllib3-1.22
Installed packages look good:
$ pip list
Package Version
----------------- ------------
asn1crypto 0.23.0
certifi 2017.11.5
cffi 1.11.2
chardet 3.0.4
cryptography 2.1.4
idna 2.6
koji 1.14.0.post1
pip 9.0.1
pycparser 2.18
pycurl 7.43.0.1
pykerberos 1.1.14
pyOpenSSL 17.5.0
python-dateutil 2.6.1
requests 2.18.4
requests-kerberos 0.11.0
rpm-py-installer 0.5.0
rpm-python 4.11.3
setuptools 38.2.4
six 1.11.0
spam 0.1
urllib3 1.22
wheel 0.30.0
The downside of this method is the additional overhead you get in maintaining the fork until the setup script is merged into upstream. This includes testing and eventually adapting koji's setup.py in your fork each time you want to sync the upstream updates. I would probably create a separate branch with the setup script committed there, sync the fork as usual and then rebase the branch on top of fork's master, but if you are used to another update strategy, stick to it.
Use koji package from TestPyPI
Actually, I found some koji wheels of the most recent version on TestPyPI. This is also the place where I got the setup.py for the fork above - I downloaded the source tar, unpacked it and copied the setup script. This means that the koji devs are looking into distributing the project via PyPI and are working on the setup script, but didn't commit it yet. While they are working on it, you can use the testing package index as the workaround. This way, you will not build the package from sources, taking the wheel instead that koji devs built and uploaded:
setup(
...
dependency_links=['https://testpypi.python.org/pypi/koji'],
install_requires=['koji'],
)
The downsides of this method are:
You don't know if the koji package from TestPyPI is installable at all. Even if it is, there's no guarantee that the installed code will work as intended (although it should). When you have the fork, you can always fix the setup script yourself - here you are doomed if the wheel file has errors.
Packages on TestPyPI are removed on a regular basis. From the docs:
Note: The database for TestPyPI may be periodically pruned, so it is not unusual for user accounts to be deleted.
Last note
You can of course combine the two workarounds and use both URLs in dependency_links:
setup(
...
dependency_links=[
'https://testpypi.python.org/pypi/koji',
'https://github.com/hoefling/tarball/master#egg=koji-1.14.0.post1',
],
install_requires=['koji'],
)
This way, if the package is not found on TestPyPI, it will be built from your fork.
Last note 2
You will probably need to install some additional system packages; at least for my system CentOS Linux release 7.3.1611 (Core) I had to install curl-devel to satisfy pycurl.