We Keep Coding

Pytorch version on g5.xlarge

I would like to run a g5.xlarge on AWS with pytorch.
However I have this error when I try to do something with cuda in python (for example torch(1., device="cuda")):
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA A10G GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
Here's the nvidia-smi:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.82.00 Driver Version: 470.82.00 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 |
| 0% 25C P0 55W / 300W | 1815MiB / 22731MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 10415 C /miniconda3/bin/python 1813MiB |
+-----------------------------------------------------------------------------+
Any idea? Which version of CUDA/pytorch should I use ?

The A10G GPU uses sm_86 which is natively supported in CUDA>=11.1.
The PyTorch website provides a very handy tool to find the right install command for any required OS / PyTorch / CUDA configuration:
For the AWS g5.xlarge instances with A10G GPUs the following configuration works for me:
Ubuntu 18.04
Python 3.8.5
PyTorch 1.12.1
CUDA 11.6
Conda install command:
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge
Pip install command:
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116

Have a look at this thread discussion thread pytorch-cuda-arch issue. The issue seems similar.

Anaconda - Install blpapi in environment

I have py2.7 as my original Anaconda install, and have installed py3.4 in an environment named py3.
I cannot seem to be able to install blpapi, the Bloomberg API library, in the py3 environment.
Once I activate py3, conda install blpapi cannot find the package, whereas pip install blpapi tells me that:
Requirement already satisfied: blpapi in C:\users\pythonic\programs\anaconda\lib\site-packages
Because it finds the package installed for python 2.7
If I try to install via the executable from here it tells me that it cannot find python 3.4 in the registry.
How do I install this package?
Edit
Apparently the DSM channel does not have the py3.5 version anymore but this channel does
https://anaconda.org/macinv/blpapi/files

One solution would be creating a Python 3.5 environment:
conda create -n py35 python=3.5
activating it:
activate py35
and installing blpapi form the channel dsm:
conda install -c dsm blpapi
The channel dsm is an Anaconda channel. It provides these files:
win-64/blpapi-3.9.0-py35_0.tar.bz2
win-64/blpapi-3.5.5-py27_0.tar.bz2
linux-64/blpapi-3.5.5-py27_0.tar.bz2
This means blpapi is available for Windows for Python 2.7 and 3.5 but not for 3.4.
To find a package, enter the package name in the Anaconda search window (top of page).

In addition to Mike Müller's answer above, I thought I would add a section on the anaconda search utility for finding packages hosted on https://anaconda.org/.
$ anaconda search blpapi
Using Anaconda API: https://api.anaconda.org
Packages:
Name | Version | Package Types | Platforms | Builds
------------------------- | ------ | --------------- | --------------- | ----------
conda-forge/blpapi | 3.9.2 | conda | linux-64, win-64, osx-64 | py27h2d50403_0, py36he980bc4_0, py36h2d50403_0, py27hdc96acc_0
: Python SDK for Bloomberg BLPAPI (<=3.9)
dsm/blpapi | 3.9.0 | conda | linux-64, win-64 | py36_0, py27_0
josh/blpapi | 3.5.5 | conda | linux-64 | py27_0
macinv/blpapi | 3.9.0 | conda | linux-64, win-64 | py36_0, py27_0, py35_0, py34_0
mbonix/blpapi | 3.9.0 | conda | win-64 | py36_0
: Bloomberg's Open Market Data Initiative is part of the company's ongoing effort to foster open solutions for the financial services industry.
p-vg/blpapi | 3.9.2 | conda | linux-64, win-64 | py36h6538335_0, py27hc56fc5f_0, py36hf484d3e_0, py27hf484d3e_0
: interface for Bloomberg API services using the Python programming language
Found 6 packages
Depending on the version of python and platform you are interested in, there are a varierty of options available.

ipcluster3.exe failed to create process

I am trying to run ipcluster3.exe on an Anaconda python 3.3 installation on Windows 7 64-bit.
However, when I run it I get an error:
C:\Anaconda\envs\py33\Scripts>ipcluster3.exe
failed to create process.
C:\Anaconda\envs\py33\Scripts>ipcluster3.exe start -n 4
failed to create process.
The error doesn't really give me enough to start debugging.

It looks like my copy of Anacondas came with a Beta version of IPython.
I updated it using Conda.
conda update -n py33 ipython
The following packages will be downloaded:
package | build
---------------------------|-----------------
ipython-2.3.1 | py33_0 3.0 MB
pyreadline-2.0 | py33_0 135 KB
------------------------------------------------------------
Total: 3.1 MB
The following packages will be UN-linked:
package | build
---------------------------|-----------------
ipython-0.13.2 | py33_0
pyreadline-2.0.dev | py33_0
The following packages will be linked:
package | build
---------------------------|-----------------
ipython-2.3.1 | py33_0 hard-link
pyreadline-2.0 | py33_0 hard-link

Anaconda/MacOS: Change default python version

I was using python with anaconda on osx 10.8 when spyder crashed. When I tried to restart it, the launcher showed it as uninstalled. I figured there might be something wrong with anaconda, so I restarted my computer, but the problem persisted.
Looking into it, I noticed the default version of python had been changed:
$ python --version
Python 3.4.1 :: Continuum Analytics, Inc.
I tried changing it back by using Apple's defaults write, relinking python with ln -sf, simply setting an alias python=python2.7, to no effect.
Then I tried removing python3 with conda, but conda remove python3 won't do the trick. Searching for packages produces this:
$ conda search python
Fetching package metadata: ..
(...)
python 1.0.1 0 defaults
(...)
. 2.7.5 2 defaults
. 2.7.5 3 defaults
(...)
* 3.4.1 0 defaults
I also looked up Continuum docs, and they recommend leaving 2.7 as standard and using an Anaconda environment if I want to use a different version, which doesn't help me.
Does anybody have any idea of how to change default version back to 2.7?
(Specs are: anaconda 1.7.0, osx 10.8.5, conda 3.5.2)

Alright, I found a similar problem in the Continuum mailing list.
This was solved by re-installing python through conda
$ conda install python=2.7
Fetching package metadata: ..
Solving package specifications: .
Package plan for installation in environment /Users/kadu/anaconda:
The following packages will be downloaded:
package | build
---------------------------|-----------------
conda-3.5.2 | py27_0 135 KB
pycosat-0.6.1 | py27_0 57 KB
python-2.7.6 | 2 16.5 MB
pyyaml-3.11 | py27_0 149 KB
requests-2.3.0 | py27_0 564 KB
------------------------------------------------------------
Total: 17.4 MB
The following packages will be UN-linked:
package | build
---------------------------|-----------------
conda-3.5.2 | py34_0
pycosat-0.6.1 | py34_0
python-3.4.1 | 0
pyyaml-3.11 | py34_0
requests-2.3.0 | py34_0
The following packages will be linked:
package | build
---------------------------|-----------------
conda-3.5.2 | py27_0 hard-link
pycosat-0.6.1 | py27_0 hard-link
python-2.7.6 | 2 hard-link
pyyaml-3.11 | py27_0 hard-link
requests-2.3.0 | py27_0 hard-link
Proceed ([y]/n)?
This can also be used to change which is the default package under anaconda environments:
$ conda install python=3.4
Fetching package metadata: ..
Solving package specifications: .
Package plan for installation in environment /Users/kadu/anaconda:
The following packages will be UN-linked:
package | build
---------------------------|-----------------
conda-3.5.2 | py27_0
pycosat-0.6.1 | py27_0
python-2.7.6 | 2
pyyaml-3.11 | py27_0
requests-2.3.0 | py27_0
The following packages will be linked:
package | build
---------------------------|-----------------
conda-3.5.2 | py34_0 hard-link
pycosat-0.6.1 | py34_0 hard-link
python-3.4.1 | 0 hard-link
pyyaml-3.11 | py34_0 hard-link
requests-2.3.0 | py34_0 hard-link
Proceed ([y]/n)?
However, this would require manually changing the builds to any other packages you use. I had to reinstall spyder, which seems to have caused the bug in the first place and was presumably reinstalled when I tried to launch it under the wrong version. All the other libraries, which I didn't try to reinstall, are still working fine under 2.7.

lxml runtime error: Reason: Incompatible library version: etree.so requires version 12.0.0 or later, but libxml2.2.dylib provides version 10.0.0

I have a perplexing problem.
I have used mac version 10.9, anaconda 3.4.1, python 2.7.6.
Developing web application with python-amazon-product-api. i have overcome an obstacle about installing lxml, referencing clang error: unknown argument: '-mno-fused-madd' (python package installation failure).
But another runtime error happened.
Here is the output from webbrowser.
Exception Type: ImportError
Exception Value:
dlopen(/Users/User_Name/Documents/App_Name/lib/python2.7/site-packages/lxml/etree.so, 2): Library not loaded: libxml2.2.dylib
Referenced from: /Users/User_Name/Documents/App_Name/lib/python2.7/site-packages/lxml/etree.so
Reason: Incompatible library version: etree.so requires version 12.0.0 or later, but libxml2.2.dylib provides version 10.0.0
I'm not sure how to proceed and have searched here and elsewhere for this particular error.

This worked for me:
brew install libxml2
brew install libxslt
brew link libxml2 --force
brew link libxslt --force

If using conda, force a reinstall of lxml:
$ conda install -f lxml
Note that forcing a reinstall may have unforeseen consequences.
I first tried conda update lxml to no effect.
Then tried conda install lxml also to no effect (but that was a while ago and ymmv -- see comment).

I was having this same issue. I realized that during pip install for my web app, lxml was attempting the following:
"Building against libxml2/libxslt in the following directory: /Users/[me]/anaconda/lib"
Not 100% sure why, but once I removed the anaconda/bin from my system path in bash_profile (anaconda sticks it in there when installing) I was able to pip install lxml correctly. After which you should be able to re-add anaconda/bin to your system path without issue.

I get this to work by doing the following:
conda install libxml2

Consider using an alternative channel in conda.
$ anaconda search -t conda pyquery
Using anaconda-server api site https://api.anaconda.org
Run 'anaconda show <USER/PACKAGE>' to get more details:
Packages:
Name | Version | Package Types | Platforms
------------------------- | ------ | --------------- | ---------------
CS109/pyquery | 1.2.9 | conda | linux-64, win-32, win-64, linux-32, osx-64
: A jquery-like library for python
asmeurer/pyquery | 1.2.6 | conda | osx-64
: https://github.com/gawel/pyquery
auto/pyquery | 1.2.8 | conda | linux-64, linux-32, osx-64
: https://github.com/gawel/pyquery
dan_blanchard/pyquery | 1.2.6 | conda | linux-64
: https://github.com/gawel/pyquery
hargup/pyquery | | conda | None-None, linux-64
: A jquery-like library for python
meloncholy/pyquery | 1.2.9 | conda | linux-64
: A jquery-like library for python
mhworth/pyquery | 1.2.8 | conda | linux-64, osx-64
: A jquery-like library for python
nbsantos/pyquery | 1.2.9 | conda | osx-64
: A jquery-like library for python
pdrops/pyquery | 1.2.8 | conda | linux-64, osx-64
: A jquery-like library for python
ziebel/pyquery | 1.2.9 | conda | linux-64
: A jquery-like library for python
Found 10 packages
Then, picking a channel, do:
conda install pyquery -c CS109

I tried almost all approaches above and failed. What worked for me was commenting out the
export DYLD_LIBRARY_PATH=/usr/lib
in my bash profile. This line is added by the anaconda installer automatically.