In a python project, I'm using conda and poetry together. conda is used to create the environment and to install poetry and python. The project dependencies are mainly managed using poetry.
However, there are some packages which I would like to install using conda as it is more convenient. Such examples include spyder, ipykernel and packages using CUDA.
The problem is that these packages installed by conda of course install dependencies themselves. If I install some other packages using poetry afterwards, these dependencies might get replaced by other versions which are again incompatible with the packages installed by conda.
Is there any way to have common dependencies which are used by both conda and poetry?
Listing all dependencies of all children recursively is not an option!
Related
I am using a conda environment to install a package and this package have dependencies that's not available in conda, so I have to use pip to install some additional packages in the conda environment. After I did all these:
I tested both:
pip list
and
conda list
And found that some dependencies occur in pip list but not in conda list. Is this OK? Do the packages installed by pip in conda enviroment also effect in this envorment?
Yes, I use a combination of pip install and conda install when setting up the environment for a project I'm working on. It works fine.
However, it is documented here that this combination can lead to issues: https://www.anaconda.com/blog/using-pip-in-a-conda-environment
According to that doc, you ought to first use conda to install as many of your packages as possible, then use pip to install the rest afterwards.
I want to install a library called "Scrapy" for Python on Windows. The developers recommend installing the package using Anaconda instead of Python3. However, will i be able to use the package in Python3 or will i need to use Anaconda for Web Scraping?
Anaconda is loved because it simplify package management and deployment in Python(and R, Ruby, Lua, Scala, Java, JavaScript, C/ C++, FORTRAN)
You get to keep your environment(program dependency) organized when using Anaconda.
With Anaconda, you can use any Python version, and libraries you need for a specific task. A workflow would be as follow(Assuming you have install Anaconda, and its available on your system path):
conda create -n scrap python=3.6
conda activate scrap
conda install -c conda-forge scrapy
Here we create environment called scrap with python version 3.6. We then activate it and install scrapy fron a conda channel forge.
While in this environment(scrap), you have access to Python 3.6 and scrapy. The best thing about this is that this is separate from your other Python and packages.
To get out of your environment,
conda deactivate
While in your environment, you can use both conda and pip to install packages to that environment. Always try finding packages in conda(plus it’s channels) before using pip, becsusr conda will check for packages compatabilities before installation. It will know which packages to upgrade or downgrade to avoid conflicts.
In few cases where a package is not in conda, then use pip.
Read more:
https://conda.io/docs/index.html
Anaconda is a python distribution which contains additional packages to the python it ships with. To have lightweight version of python (without many additional packages), you can install Miniconda. Anaconda and Miniconda come with the conda package manager. You can use it to install and update packages specific to its python distribution.
To install scrapy package using Anaconda / Miniconda, in the Windows Command Prompt simply type in:
conda install scrapy
In Anaconda you can install almost all python packages using conda install or pip install.
You have to goto Anaconda prompt and type in pip install Scrapy
Anaconda just simplified the installation, but you can also install scrapy through PyPi as
pip install scrapy
When I install a package through pip (since it was not available on Anaconda), it also pulls all dependencies. It seems it will use the pip versions of the dependencies, even if conda versions (same name) are available.
How can I easily install a pip package, but use conda for the dependencies where such a package exists?
There is no easy way, I suspect. Create a virtual environment, install all anticipated dependencies using conda and then install the main package using pip without -U/--upgrade. pip seeing dependencies installed will not install them again.
I have installed a fresh anaconda v4.4. I realized that python packages can be installed using both conda and pip. What is the effect of using pip to install python packages instead of conda when using anaconda? Will the pip-installed libraries cease to function? I am using python v3
EDIT: I don't think the question is a duplicate of What is the difference between pip and conda?
That question explains the difference between pip and conda but does not talk about the effect of using pip when conda can be used.
Everything might keep working if you use pip to install vs conda. However, Conda cannot manage dependencies that pip has installed - it cannot upgrade them, or remove them. More importantly, conda will install a package even if its already been installed with pip! Try this test:
conda create -n testenv python=3
conda activate testenv
pip install numpy
conda install scipy
You will see from the third command that conda will want to re-install NumPy, even though it has already been installed with pip. This can cause problems if there are C libraries whose linking is different, or something like that. In general, whenever possible, use conda to install packages into conda environments.
When using the Anacoda Python distribution, what is the best way to install a PyPi package that isn't available directly through Anaconda? For now I'm using:
conda pipbuild [pypi_name]
conda install --use-local [package_spec]
But I'm unclear if this is the best way and if conda update --all will update these packages when updates are made available. I'm also unclear what the point of binstar is when PyPi already exists.
I will disagree with the accepted response and note that pip install [some-pypi-package] is often the best way to install PyPi packages in Conda environments.
While the packages won't be managed by the Conda package manager, they will still be managed by the Anaconda environment. It will download the correct version of the package for the active Python install and update it correctly using the pip package manager.
When using Anaconda, you should turn to conda before pip when you can, but you don't lose any of the replicability benefits of using Anaconda when you use pip.
Anaconda recently published a doc that supports this: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#using-pip-in-an-environment
If you want to build conda packages for PyPI packages, the recommended way is to use conda skeleton pypi package and use conda build package on the recipe that it creates. To install the package, use conda install --use-local package (here and elsewhere, package is the name of the PyPI package you wish to install).
You will need to update the recipe each time the package is updated.
You can also use pip to install these packages. There are two disadvantages: firstly, these packages won't be managed by conda at all. Secondly, these packages will not work if your default python version is different from the python version you are using in conda.
Since version 4.6.0, Conda has improved its interoperability with pip:
Conda and pip have historically had difficulties getting along. Pip
hasn’t respected Conda’s environment constraints, while Conda has been
all too happy to clobber pip-installed software. It’s a mess. Conda
4.6.0 adds preview support for better interoperability. With this interoperability, Conda can use pip-installed packages to satisfy
dependencies, and can even remove pip-installed software cleanly and
replace them with Conda packages when appropriate. There’s still room
for improvement before pip and Conda are hunky-dory BFFs, but we hope
this is a good start. This feature is disabled by default right now
because it can significantly impact Conda’s performance. If you’d like
to try it, you can set this condarc setting:
conda config --set pip_interop_enabled True
So, the way to get PyPI packages into conda (at the time of writing this) seems to be:
pip install <package>
If you want conda to replace the PyPI packages with its own (where possible), just run:
conda update --all
Given that the above setting is made. Conda marks its own channels as higher priority than pip, thus packages will be replaced.
There is a caveat (thanks #alfalfasprout): Since conda did not install the pypi packages they are not included in conda export --from-history. You'd have to get the list (from conda or pip) and install these packages separately via pip, like you did originally. Thus, for people using the conda history functionality, there is at least a second step needed.