We Keep Coding

How to clean up anaconda base environment?

I have just reinstalled Miniconda. After that I ran pip list in base environment. The output is following:
Package Version
---------------------- ---------
brotlipy 0.7.0
certifi 2021.10.8
cffi 1.15.0
charset-normalizer 2.0.4
colorama 0.4.4
conda 4.12.0
conda-content-trust 0+unknown
conda-package-handling 1.8.1
cryptography 36.0.0
idna 3.3
menuinst 1.4.18
pip 21.2.4
pycosat 0.6.3
pycparser 2.21
pyOpenSSL 22.0.0
PySocks 1.7.1
pywin32 302
requests 2.27.1
ruamel-yaml-conda 0.15.100
setuptools 61.2.0
six 1.16.0
tqdm 4.63.0
urllib3 1.26.8
wheel 0.37.1
win-inet-pton 1.1.0
wincertstore 0.2
But the fact is that I did not install them. How packages like tqdm, colorama, brotlipy, cryptography and others appeared here? It supposed to be an empty base environment. Your suggestions?

Do:
conda uninstall -n base --all

i would do:
conda clean -a -d
#-d = dryrun
and if that's okay with me
conda clean -a -y
#-y = yes (no promt)

Tensorflow Object Detection Api M1 Macbook Conflict Error

Machine: MacBook Air M1 2020
OS: macOs BigSur 11.4
Python version of venv: Python 3.8.6
Tensorflow version: ATF Apple Tensorflow 0.1a3
Pip version: 21.2.4
I have installed Tensorflow from github using this guide.
Now, my pip list is this.
Package Version
----------------------- ---------
absl-py 0.13.0
appnope 0.1.2
astunparse 1.6.3
backcall 0.2.0
cached-property 1.5.2
cachetools 4.2.2
certifi 2021.5.30
charset-normalizer 2.0.4
cycler 0.10.0
Cython 0.29.24
debugpy 1.4.1
decorator 5.0.9
entrypoints 0.3
flatbuffers 2.0
gast 0.5.2
google-auth 1.35.0
google-auth-oauthlib 0.4.5
google-pasta 0.2.0
grpcio 1.33.2
h5py 2.10.0
idna 3.2
ipykernel 6.2.0
ipython 7.26.0
ipython-genutils 0.2.0
jedi 0.18.0
jupyter-client 7.0.1
jupyter-core 4.7.1
Keras-Preprocessing 1.1.2
kiwisolver 1.3.1
Markdown 3.3.4
matplotlib 3.4.3
matplotlib-inline 0.1.2
nest-asyncio 1.5.1
numpy 1.18.5
oauthlib 3.1.1
opt-einsum 3.3.0
packaging 21.0
parso 0.8.2
pexpect 4.8.0
pickleshare 0.7.5
Pillow 8.3.1
pip 21.2.4
prompt-toolkit 3.0.20
protobuf 3.17.3
ptyprocess 0.7.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
Pygments 2.10.0
pyparsing 2.4.7
python-dateutil 2.8.2
pyzmq 22.2.1
requests 2.26.0
requests-oauthlib 1.3.0
rsa 4.7.2
setuptools 57.4.0
six 1.16.0
tensorboard 2.6.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.0
tensorflow-addons 0.1a3
tensorflow-estimator 2.6.0
tensorflow-hub 0.12.0
tensorflow 0.1a3
termcolor 1.1.0
tornado 6.1
traitlets 5.0.5
typeguard 2.12.1
typing-extensions 3.10.0.0
urllib3 1.26.6
wcwidth 0.2.5
Werkzeug 2.0.1
wheel 0.37.0
wrapt 1.12.1
I want install Object Detection Api from Tensorflow in that link.
I cloned the repo and them I follow the guide. (Python Package Installation)
When I execute this command
python -m pip install --use-feature=2020-resolver .
It starts to download, and start a print very long errors.
At the end of the operations, it gives me this error.
Using cached scipy-1.2.3.tar.gz (23.3 MB)
Collecting pandas
Using cached pandas-1.3.2-cp38-cp38-macosx_11_0_arm64.whl
Collecting tf-models-official>=2.5.1
Using cached tf_models_official-2.6.0-py2.py3-none-any.whl (1.8 MB)
Collecting kaggle>=1.3.9
Using cached kaggle-1.5.12-py3-none-any.whl
Collecting py-cpuinfo>=3.3.0
Using cached py_cpuinfo-8.0.0-py3-none-any.whl
Requirement already satisfied: numpy>=1.15.4 in /Users/stefan/Desktop/Studio/TFOD/tf-m1/lib/python3.8/site-packages (from tf-models-official>=2.5.1->object-detection==0.1) (1.18.5)
Collecting opencv-python-headless
Using cached opencv_python_headless-4.5.3.56-cp38-cp38-macosx_11_0_arm64.whl (10.7 MB)
Collecting tf-models-official>=2.5.1
Using cached tf_models_official-2.5.1-py2.py3-none-any.whl (1.6 MB)
Collecting tensorflow-datasets
Using cached tensorflow_datasets-4.4.0-py3-none-any.whl (4.0 MB)
Collecting google-api-python-client>=1.6.7
Downloading google_api_python_client-2.18.0-py2.py3-none-any.whl (7.4 MB)
|████████████████████████████████| 7.4 MB 3.4 MB/s
Collecting oauth2client
Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB)
Collecting tensorflow-model-optimization>=0.4.1
Using cached tensorflow_model_optimization-0.6.0-py2.py3-none-any.whl (211 kB)
Collecting pyyaml>=5.1
Downloading PyYAML-5.4.1.tar.gz (175 kB)
|████████████████████████████████| 175 kB 31.3 MB/s
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Collecting gin-config
Using cached gin_config-0.4.0-py2.py3-none-any.whl (46 kB)
Collecting sacrebleu
Using cached sacrebleu-2.0.0-py3-none-any.whl (90 kB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of object-detection to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install object-detection because these package versions have conflicting dependencies.
The conflict is caused by:
tf-models-official 2.6.0 depends on tensorflow-text>=2.5.0
tf-models-official 2.5.1 depends on tensorflow-addons
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

I have the same issue installing the Object Detection API for Tensorflow 2 (OD API) from sources on my MacBook Air M1 2020. It starts to lookup/download all available dependencies with very long errors and after several hours the process drains all available RAM and forces the laptop to reboot. I think the problem is with incompatible dependencies for arm64. I tried to build/install OD API for Tensorflow 1 instead and it worked! I successfully trained a model with TensorFlow 2 and GPU enabled.
Use the tf1 folder when you installing the OD API instead of tf2:
cd models/research
# Compile protos.
protoc object_detection/protos/*.proto --python_out=.
# Install TensorFlow Object Detection API.
cp object_detection/packages/tf1/setup.py .
python -m pip install --use-feature=2020-resolver .
or just use this guide for installing OD API: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1.md
By the way,
here is a working Tensorflow setup on Apple M1 silicon with the latest TensorFlow versions and Metal GPU acceleration: https://github.com/ctrahey/m1-tensorflow-config
the best guide for object detection: https://neptune.ai/blog/how-to-train-your-own-object-detector-using-tensorflow-object-detection-api

I successfully install it with.
python -m pip install --force --no-dependencies .
My list of commands for install correctly tf2.0 for m1
conda create —-name=tf-m1
conda activate tf-m1
conda install python=3.8.6 -y
sh Desktop/PATH TO GITHUB DIR OF TENSORFLOW MAC(i used 0.1a3)/install_venv.sh /Users/stefan/miniforge3/envs/tf-m1
python -m pip install --upgrade pip
pip install ipykernel jupyter
python -m ipykernel install --user --name=tensorflow-m1.0
Tensorflow Test : ok (import tensorflow as tf; print(tf.__version__))
NOW USE CONDA INSTALL
conda install -c conda-forge matplotlib -y
conda install -c conda-forge scikit-learn -y
conda install -c conda-forge opencv -y
conda install -c conda-forge pandas -y
Tensorflow Test : ok
cd Desktop/PATH/
mkdir -p Tensorflow/models
git clone https://github.com/tensorflow/models Tensorflow/models
cd Tensorflow/models/research && protoc object_detection/protos/*.proto --python_out=. && cp object_detection/packages/tf2/setup.py . && python -m pip install --force --no-dependencies .
Object detection api have some dependencies i had installed.
(Pyarrow and apache-beam are not supported at the moment, but I think this isn't essential for general working of api)
pip install tf-slim
pip install pycocotools
pip install lxml
pip install lvis
pip install contextlib2
pip install --no-dependencies tf-models-official
pip install avro-python3
pip install pyyaml
Pip install gin-config
I don't know if is it the perfect installation of Tensorflow and TensorFlow object-detection-api, but at the moment this worked for me.

Things should work better if you upgrade to OS Monterey and install conda from miniforge and the packages listed below.
As of Oct. 25, 2021 macOS 12 Monterey is generally available.
Upgrade your machine to Monterey.
If you have conda installed, uninstall it.
Then follow the instructions from Apple here.
Cleaned up below:
Download and install Conda from Miniforge:
chmod +x ~/Downloads/Miniforge3-MacOSX-arm64.sh
sh ~/Downloads/Miniforge3-MacOSX-arm64.sh
source ~/miniforge3/bin/activate
In a conda environment, install the TensorFlow dependencies, base TensorFlow, and TensorFlow metal:
conda install -c apple tensorflow-deps
pip install tensorflow-macos
pip install tensorflow-metal
You should be good to go.

pip does not install all of the packages if I ask it to do it just once

I have a fresh installation of python3 and python3-pip on Ubuntu.
I invoked the command:
pip3 install ansible packaging msrestazure docker-py ansible[azure] openshift
Then I invoked pip3 list. Here's the result:
root#7e8175337b62:/# pip3 list
Package Version
------------------- ---------
adal 1.2.4
ansible 2.9.11
cachetools 4.1.1
certifi 2020.6.20
cffi 1.14.1
chardet 3.0.4
cryptography 3.0
docker-py 1.10.6
docker-pycreds 0.4.0
google-auth 1.20.0
idna 2.10
isodate 0.6.0
Jinja2 2.11.2
kubernetes 11.0.0
MarkupSafe 1.1.1
msrest 0.6.18
msrestazure 0.6.4
oauthlib 3.1.0
openshift 0.11.2
packaging 20.4
pip 20.0.2
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.20
PyJWT 1.7.1
pyparsing 2.4.7
python-dateutil 2.8.1
python-string-utils 1.0.0
PyYAML 5.3.1
requests 2.24.0
requests-oauthlib 1.3.0
rsa 4.6
ruamel.yaml 0.16.10
ruamel.yaml.clib 0.2.0
setuptools 45.2.0
six 1.15.0
urllib3 1.25.10
websocket-client 0.57.0
wheel 0.34.2
I saw that not all of the azure packages got installed. So, I run pip3 install ansible[azure] this time, although I already asked pip to install that package. After installation finishes, this is the result of pip3 list:
root#7e8175337b62:/# pip3 list
Package Version
------------------------------ ---------
adal 1.2.4
ansible 2.9.11
applicationinsights 0.11.9
argcomplete 1.12.0
azure-cli-core 2.0.35
azure-cli-nspkg 3.0.2
azure-common 1.1.11
azure-graphrbac 0.40.0
azure-keyvault 1.0.0a1
azure-mgmt-authorization 0.51.1
azure-mgmt-automation 0.1.1
azure-mgmt-batch 5.0.1
azure-mgmt-cdn 3.0.0
azure-mgmt-compute 4.4.0
azure-mgmt-containerinstance 1.4.0
azure-mgmt-containerregistry 2.0.0
azure-mgmt-containerservice 4.4.0
azure-mgmt-cosmosdb 0.5.2
azure-mgmt-devtestlabs 3.0.0
azure-mgmt-dns 2.1.0
azure-mgmt-hdinsight 0.1.0
azure-mgmt-iothub 0.7.0
azure-mgmt-keyvault 1.1.0
azure-mgmt-loganalytics 0.2.0
azure-mgmt-marketplaceordering 0.1.0
azure-mgmt-monitor 0.5.2
azure-mgmt-network 2.3.0
azure-mgmt-nspkg 2.0.0
azure-mgmt-rdbms 1.4.1
azure-mgmt-redis 5.0.0
azure-mgmt-resource 2.1.0
azure-mgmt-servicebus 0.5.3
azure-mgmt-sql 0.10.0
azure-mgmt-storage 3.1.0
azure-mgmt-trafficmanager 0.50.0
azure-mgmt-web 0.41.0
azure-nspkg 2.0.0
azure-storage 0.35.1
bcrypt 3.1.7
cachetools 4.1.1
certifi 2020.6.20
cffi 1.14.1
chardet 3.0.4
colorama 0.4.3
cryptography 3.0
docker-py 1.10.6
docker-pycreds 0.4.0
google-auth 1.20.0
humanfriendly 8.2
idna 2.10
isodate 0.6.0
Jinja2 2.11.2
jmespath 0.10.0
knack 0.3.3
kubernetes 11.0.0
MarkupSafe 1.1.1
msrest 0.6.1
msrestazure 0.5.0
oauthlib 3.1.0
openshift 0.11.2
packaging 20.4
paramiko 2.7.1
pip 20.0.2
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.20
Pygments 2.6.1
PyJWT 1.7.1
PyNaCl 1.4.0
pyOpenSSL 19.1.0
pyparsing 2.4.7
python-dateutil 2.8.1
python-string-utils 1.0.0
PyYAML 5.3.1
requests 2.24.0
requests-oauthlib 1.3.0
rsa 4.6
ruamel.yaml 0.16.10
ruamel.yaml.clib 0.2.0
setuptools 45.2.0
six 1.15.0
tabulate 0.8.2
urllib3 1.25.10
websocket-client 0.57.0
wheel 0.30.0
xmltodict 0.12.0
As you can see, the second time I requested the installation of ansible[azure] it actually installed all the packages. Why didn't it work the first time I asked?

1- pip3 install packaging ansible msrestazure docker-py ansible[azure] openshift
2- pip3 install packaging msrestazure docker-py ansible[azure] openshift
Why the first one install all the azure packages? but the second is just only installing ansible?
The reason is, when ansible[azure] is executed, pip checks whether the ansible package is installed or not. The ansible package is installed in the first place (pip3 install ansible). Therefore the time we execute pip3 install ansible[azure], pip thought all the necessary packages were installed. Therefore skips the remaining and the crucial azure packages.
Since this is a long explanation I couldn't fit into the comment section. Therefore, I wrote as an answer.

How to deal with new pip3.x for each Python version

After recently upgrading from python 3.7 to 3.8, I realized that my packages installed on my system were not transferred over after installing 3.8. Specifically, I am seeing the below for pip3.7 list:
Package Version
----------------- ---------
appdirs 1.4.3
astroid 2.3.3
attrs 19.3.0
click 7.1.1
isort 4.3.21
lazy-object-proxy 1.4.3
mccabe 0.6.1
pathspec 0.7.0
pip 20.0.2
pylint 2.4.4
regex 2020.2.20
setuptools 46.0.0
six 1.14.0
toml 0.10.0
typed-ast 1.4.1
wheel 0.34.2
wrapt 1.11.2
while pip3 or pip3.8 list shows:
Package Version
---------- -------
pip 20.0.2
setuptools 41.2.0
Is this the expected behavior or did not do something wrong while upgrading to 3.8? If this is the expected behavior, how do you transfer the packages over to the new version? Is it not best practice to do so?

Can't install any pip package within my docker environment, since it won't be recognized

I am running a Cookiecutter django project in a docker environment and I would like to add new packages via pip. Specifically I want to add: djangorestframework-jwt
When I do:
docker-compose -f local.yml run --rm django pip install
it seems like it would be perfectly working because I get:
Successfully installed PyJWT-1.7.1 djangorestframework-jwt-1.11.0
Now the problem is that it doesn't install it. It doesn't appear when I run pip freeze, and also not in pip list
Then I tried to put it into my requirements.txt file and run it with:
docker-compose -f local.yml run --rm django pip install -r requirements/base.txt
Same result. It says that it is successfully installed but it is not. I thought it might be a problem with my django version and the package, but the same happens when I try to update my pip. It says it updated, but when I run pip install -upgrade pip I get again:
You should consider upgrading via the 'pip install --upgrade pip' command.
I'm running out of options.
My requirements:
-r ./base.txt
Werkzeug==0.14.1 # https://github.com/pallets/werkzeug
ipdb==0.11 # https://github.com/gotcha/ipdb
Sphinx==1.7.5 # https://github.com/sphinx-doc/sphinx
psycopg2==2.7.4 --no-binary psycopg2 # https://github.com/psycopg/psycopg2
# Testing
# ------------------------------------------------------------------------------
pytest==3.6.3 # https://github.com/pytest-dev/pytest
pytest-sugar==0.9.1 # https://github.com/Frozenball/pytest-sugar
# Code quality
# ------------------------------------------------------------------------------
flake8==3.5.0 # https://github.com/PyCQA/flake8
coverage==4.5.1 # https://github.com/nedbat/coveragepy
# Django
# ------------------------------------------------------------------------------
factory-boy==2.11.1 # https://github.com/FactoryBoy/factory_boy
django-debug-toolbar==1.9.1 # https://github.com/jazzband/django-debug-toolbar
django-extensions==2.0.7 # https://github.com/django-extensions/django-extensions
django-coverage-plugin==1.5.0 # https://github.com/nedbat/django_coverage_plugin
pytest-django==3.3.2 # https://github.com/pytest-dev/pytest-django
djangorestframework-jwt==1.11.0 # https://github.com/GetBlimp/django-rest-framework-jwt
Output of pip list:
Package Version
------------------------ --------
alabaster 0.7.12
argon2-cffi 18.1.0
atomicwrites 1.3.0
attrs 19.1.0
Babel 2.6.0
backcall 0.1.0
certifi 2019.3.9
cffi 1.12.2
chardet 3.0.4
coreapi 2.3.3
coreschema 0.0.4
coverage 4.5.1
decorator 4.4.0
defusedxml 0.5.0
Django 2.0.7
django-allauth 0.36.0
django-coverage-plugin 1.5.0
django-crispy-forms 1.7.2
django-debug-toolbar 1.9.1
django-environ 0.4.5
django-extensions 2.0.7
django-model-utils 3.1.2
django-redis 4.9.0
django-widget-tweaks 1.4.3
djangorestframework 3.8.2
docutils 0.14
factory-boy 2.11.1
Faker 1.0.4
flake8 3.5.0
idna 2.8
imagesize 1.1.0
ipdb 0.11
ipython 7.4.0
ipython-genutils 0.2.0
itypes 1.1.0
jedi 0.13.3
Jinja2 2.10
MarkupSafe 1.1.1
mccabe 0.6.1
more-itertools 6.0.0
oauthlib 3.0.1
packaging 19.0
parso 0.3.4
pexpect 4.6.0
pickleshare 0.7.5
Pillow 5.2.0
pip 19.0.3
pluggy 0.6.0
prompt-toolkit 2.0.9
psycopg2 2.7.4
ptyprocess 0.6.0
py 1.8.0
pycodestyle 2.3.1
pycparser 2.19
pyflakes 1.6.0
Pygments 2.3.1
pyparsing 2.3.1
pytest 3.6.3
pytest-django 3.3.2
pytest-sugar 0.9.1
python-dateutil 2.8.0
python-slugify 1.2.5
python3-openid 3.1.0
pytz 2018.5
redis 3.2.1
requests 2.21.0
requests-oauthlib 1.2.0
setuptools 40.8.0
six 1.12.0
snowballstemmer 1.2.1
Sphinx 1.7.5
sphinxcontrib-websupport 1.1.0
sqlparse 0.3.0
termcolor 1.1.0
text-unidecode 1.2
traitlets 4.3.2
Unidecode 1.0.23
uritemplate 3.0.0
urllib3 1.24.1
wcwidth 0.1.7
Werkzeug 0.14.1
wheel 0.33.1
Any help is highly appreciated! Thanks...

docker-compose run starts a new container and executes the command in it. When used with --rm flag the container gets removed after command completes.
What happens is you get a new container created, and packages installed, or pip upgraded, inside this container. Once the command completes the container is removed.
If later on you run something like docker-compose -f local.yml run --rm pip list a brand new container will get created and pip list executed inside it, showing no packages from previous run since they were installed in a different container, which is already removed.
A better way would be to create docker image that includes your application and install pip packages during docker build. You can check a sample in this question
This way any time you start a container from your image it will have all packages inside.