My aim to to explore GPU acceleration for tabular data with 10,000 to 10M+ records. I am most familiar with Pandas, so cuDF seems like a good place to start.
I'm finding mixed results re: whether cuDF will run on my system (Windows 7 Pro 64-bit, i7-6820HQ, 32GB RAM, NVidia Quadro M2000M 4GB). There is also an onboard graphics card.
per the gitHub page (https://github.com/rapidsai/cudf):
CUDA/GPU Requirements
CUDA 10.0+ (YES - I have v10.1.120)
NVIDIA driver 410.48+ (YES - I have 432.06)
Pascal architecture or better (NO - Maxwell)
I have heard that Pascal architecture is preferred/optimal as opposed to a requirement, but maybe that was for older versions of cuDF? Just this morning I heard it will run on Win 64, though performance benefits may also be reduced. Nonetheless, I'm interested in giving it a shot.
When I install from the conda prompt (python 3.6 env) using the recommended command for my CUDA version:
conda install -c rapidsai -c nvidia -c numba -c conda-forge cudf=0.13
python=3.6 cudatoolkit=10.1
I get:
Collecting package metadata (repodata.json): done Solving environment:
failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from
current channels:
cudf=0.13
Current channels:
https://conda.anaconda.org/rapidsai/win-64
https://conda.anaconda.org/rapidsai/noarch
https://conda.anaconda.org/nvidia/win-64
https://conda.anaconda.org/nvidia/noarch
https://conda.anaconda.org/numba/win-64
https://conda.anaconda.org/numba/noarch
https://conda.anaconda.org/conda-forge/win
https://conda.anaconda.org/conda-forge/noa
https://repo.anaconda.com/pkgs/main/win-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/win-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-6
https://repo.anaconda.com/pkgs/msys2/noarc
To search for alternate channels that may provide the conda package
you're looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
When I go to anaconda.org and search for cuDF (or RAPIDS), all I find are Linux installs.
I attended an Anaconda-sponsored webinar earlier today where the speaker said it'll run in Win-64, though this older post suggest maybe I need to build from source:
Package not found error while installing CuSpatial or CuDf library
I'm not ready to attempt a build from source. Am I just wasting my time? Recommendations appreciated (for either resolving cuDF with my system or alternative packages).
cuDF maintainer here.
Currently, cuDF nor any other RAPIDS libraries are supported in a native Windows environment. There's an issue tracking Windows support here: https://github.com/rapidsai/cudf/issues/28.
In general, native Windows support is not a priority for us, especially given the push towards GPU support in WSL2 that is currently in open beta.
Apparently there are some news regarding this. Here one can find the guide for using NVIDIA CUDA on Windows Subsystem for Linux.
Getting started with running CUDA on WSL requires you to complete
these steps in order:
1. Installing the latest builds from the Microsoft Windows Insider Program
2. Installing the NVIDIA preview driver for WSL 2
3. Installing WSL 2
Important note regarding the installation of the latest builds from the Microsoft Windows Insider Program
Ensure that you install Build version 20145 or higher.
You can check your build version number by running winver via the Windows Run command. (Source)
Hopefully next year a version of Windows that meets the Build version 20145 or higher requirement will be released and then one doesn't need to run an "Insider Program" build.
Source for Windows 10 release information.
Here one will be able to follow all the updates regarding the Support for Windows.
Related
I have downloaded the anaconda3 a complete newer version from official site.. I tried to create an environment but the 'solving environment' keeps on running..
I tried turning off windows defender but it didn't work.. someone plz help.. I am using windows 11 pro, and I have downloaded and installed Anaconda3-2022.10-Windows-x86_64 on 23/12/2022
The actively available Anaconda builds for Python 3.6 all use 3.6.10, so you would have a possibly easier solve with:
conda create -n ssd_env python=3.6 anaconda
However, there is the custom build, anaconda=custom=py36_1 that should be generally compatible with python>=3.6,<3.7. That is what I get when simulating the solve from OP:
CONDA_SUBDIR=win-64 mamba create -n ssd_env --override-channels -c anaconda python=3.6.8 anaconda
which solves almost instantly with Mamba.
Using conda also takes unreasonably long for me, even when explicitly identifying the anaconda=custom=py36_1. The reason this takes so long is that this anaconda package has no version constraints on the 100+ packages it specifies, which means a huge search space. This is aggravated by the fact that the solvers work from latest to oldest package versions, and the versions that are expected to be identified are ~3 years down the stack.
I recommend:
Use Mamba as the solver.
Don't use the anaconda package unless absolutely needed. Most users do not ever need all those packages - instead just specify the packages actually required.
Last I wrote a python project was less than 2 months ago and everything worked fine. I'm not sure if while working on other project I messed something up on my mac but now when trying to run python files which used to run perfectly, the following error appears:
dlopen(/opt/homebrew/lib/python3.9/site-packages/pandas/_libs/interval.cpython-39-darwin.so, 0x0002): tried: '/opt/homebrew/lib/python3.9/site-packages/pandas/_libs/interval.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/interval.cpython-39-darwin.so' (no such file), '/usr/lib/interval.cpython-39-darwin.so' (no such file)
I understand there is an issue with the architecture x86 vs arm so I tried seeing what platform the terminal is on with:
python -c 'import platform; print(platform.platform())'
which confirmed it was arm64.
Doing some googling and looking at similar issues such as Trouble installing Pandas on new MacBook Air M1 it seems like it would be possible to run the python project in an x86 environment, however like already mentioned, it worked fine before, and it seems there was no update since, so what could have happened that pandas (and perhaps other libs) no longer work on arm, and how can it be reverted?
You should try using miniforge.
its definition from its GitHub repository:
This repository holds a minimal installer for Conda specific to conda-forge. Miniforge allows you to install the conda package manager with the following features pre-configured:
Its main feature that will be useful for us
An emphasis on supporting various CPU architectures (x86_64, ppc64le, and aarch64 including Apple M1).
The Process I use:
Create a conda environment and usually go with "python3.9".
Install the packages from the conda, most of them are available but some are not.
After trying and installing all the packages possible with miniforge, I use PIP for the remaining packages.
This workflow has worked pretty well for me and hope it helps you.
I want to utilize the native m1 performance and I think you will be able to see the difference.
By default, miniforge only downloads arm compatible builds of python packages. till now I have not faced any major issue working with most data science libraries, except with PyTorch.
I'm trying to install ibm-db into a virtual environment with Python running on an IBM i. I'm running into the following exception about limits.h no such file or directory. All the other similar questions I've found were either on Linux or MacOS, so not sure how to resolve this on an IBM i.
I also tried doing this at the system level without a virtual environment and ran into the same problem. I'm trying to install ibm-db version 3.0.2 (latest at time of this question) and that could be part of the problem. If I list the system installed packages, there is a version 2.0.5.12 of ibm-db already installed.
My IBM i is at release V7R2.
Update 2021-01-15 # 1:42PM
This seems to not be limited to the latest version of ibm-db. I tried installing a few other versions, right down to 2.0.5 and I get the same error. So it sounds like something is missing in the environment.
Despite a few mentions of IBM i (or i5) in the main ibm-db project, it really doesn't have proper support for being run from IBM i. (It may be the case that connecting to IBM i from some other platform still works; I have not verified this.) If you want to use ibm-db with IBM's Python for PASE, you need to be using Kevin Adler's fork. See the issue establishing this.
The most straightforward way to do this is via the RPMs (installed via yum directly, or the Access Client Solutions GUI). It looks like you already have the latest version of this installed (2.0.5.12 as of this writing).
Further, if you want to use virtual environments (which is a good idea), your best bet is to use the --system-site-packages option, so that it finds the RPM-installed one.
Finally, if you are not tied to ibm-db, the recommended way to connect in a PEP-249-ish way is through ODBC. There is now sufficient support for ODBC on PASE that you can actually develop and test code connecting from your PC to IBM i using pyodbc, and then just use that exact same code on PASE.
I want to know whether opencv3 and python3 have GPU mode,I looked at this link and knew that there was no GPU mode when opencv2, but does opencv3 have GPU mode now?
You can manually compile the OpenCV 3 source with GPU support for Python 3. All steps are outlined in this blog post. To answer your question, follow all parts of Step 0 up to and including step 5 to install OpenCV 3 with GPU support for Python 3.
The major requirement is to have an NVIDIA graphics card with CUDA support and all required graphics drivers installed. These steps should work for any debian-like linux distro, I have tested on Ubuntu 16.04, 17.04 and Linux Mint 18.3 without problem.
As per the latest release 4.0.0-pre, GPU modules are not yet supported by OpenCV-python.
Remaining fields specify what modules are to be built. Since GPU modules are not yet supported by OpenCV-Python, you can completely
avoid it to save time (But if you work with them, keep it there).
Source: OpenCV Docs
Related Question
I tried to install a hoomd_script molecular dynamics software on my imac (it's imac pro before 2009, the system is OS X El captain v10.11.3). I have successfully compiled this to iMac, but when I import this hoomd_script in Python 2.7.12, Python crashes completely and I get the error:
Illegal instruction: 4.
I have installed all the prerequisites packages (including boost, sphinx, git, mpich2, numpy, cmake, pkg-config, sqlite) using conda.
I applied python -vc 'hoomd_script' to test, and the result is here. I tried to reinstall all the packages including conda and recompile the hoomd, but nothing changed. I wonder how can I fix this. Thanks!
As stated on the HOOMD-blue web page, the conda builds require a CPU capable of AVX instructions (2011 or newer). The illegal instruction results because you are trying to execute an instruction that your processor does not support.
Compiling hoomd from a clean build directory on your system should result in a binary that your system can execute. Note that conda provided prerequisite libraries are difficult to work with: I recommend using macports or homebrew.