I'm doing some scientific computation in python, and recently switched away from Anaconda since it doesn't yet support python 3.10 (which has some new features I'd like to use). I am now running on python version 3.10.9. I have a pipenv virtual environment set up for this, and in it I've got numpy version 1.23.5 installed. (Trying to upgrade to a newer version fails with a bunch of errors -- if upgrading numpy ends up being necessary I'll need to open a new question to ask about how to get that working.)
Everything initially seems to be working fine, until I try to run a parallelized numpy computation. At that point, I get the error DLL load failed while importing _multiarray_umath: The specified module could not be found. Here's the associated traceback:
Traceback (most recent call last):
File "C:\Users\charles\.virtualenvs\pipenv-WB8juvy1\lib\site-packages\numpy\core\__init__.py", line 23, in <module>
from . import multiarray
File "C:\Users\charles\.virtualenvs\pipenv-WB8juvy1\lib\site-packages\numpy\core\multiarray.py", line 10, in <module>
from . import overrides
File "C:\Users\charles\.virtualenvs\pipenv-WB8juvy1\lib\site-packages\numpy\core\overrides.py", line 6, in <module>
from numpy.core._multiarray_umath import (
ImportError: DLL load failed while importing _multiarray_umath: The specified module could not be found.
I can import numpy just fine, and I can use various functions from numpy without issue as well. I only get problems when I specifically want to do some parallelized computations. At that point my console gets flooded with this error (as far as I can tell, it's printing nearly exactly the same thing over and over, presumably once for every worker that it tried to launch).
The following text is also included in the error message:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.10 from "C:\Users\charles\.virtualenvs\pipenv-WB8juvy1\Scripts\python.exe"
* The NumPy version is: "1.24.1"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
While this looks pretty promising, I've read through the documentation on that page and nothing seems to be addressing the particular issue I'm running into here.
Additional details about my setup: The virtual environment was set up with pipenv, using the --site-packages flag (as I don't seem to be able to properly install PyQt5 in the virtual environment, so I'm just sourcing it from the system). I've got numpy installed in both my system's site-packages - that is version 1.24.1 (which checks out with the error messages) - and in the virtual environment's site packages (I didn't explicitly install it there, it was installed as a dependency; the virtual version is 1.23.5). Uninstalling numpy from the system site packages does not fix the issue.
I've also tried completely deleting the virtual environment and reinstalling it from the pipfile, but that also doesn't fix the issue. The version of numpy which is installed in the virtual environment does indeed have a DLL in its .lib folder, so it's not just missing. As far as I can tell there's nothing weird about my path; just in case I tried manually adding the .lib folders where the DLLs live to the path. I added both the virtual and system site packages to the path, and tried sourcing them in various different orders, to no avail.
I'm kinda at the end of my ability to troubleshoot this at this point, hence this request for help. In the meantime I'm resorting to doing all of my computations serially, but (as you might imagine) this is very slow and not very sustainable for the long term. If nobody can help me fix this, I'm going to have to just go back to Anaconda and give up on features from the newer python versions until they support them. (Or maybe I can try working without a virtual environment, but I've messed up systems pretty badly that way before and I'm kind of loathe to take that approach).
Related
In Power BI, I am trying to import a CSV-file that's stored in an AWS S3 bucket. I used the script I found here to import the data, but I keep getting this error:
Details: "ADO.NET: Python script error. <pi>C:\Users\MyName\anaconda3\envs\PowerBI\lib\site-packages\numpy\__init__.py:143: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service from . import
_distributor_init Traceback (most recent call last): File "PythonScriptWrapper.PY", line 2, in <module>
import os, pandas, matplotlib File "C:\Users\MyName\anaconda3\envs\PowerBI\lib\site-packages\pandas\__init__.py", line 17, in <module>
"Unable to import required dependencies:\n" + "\n".join(missing_dependencies) ImportError: Unable to import required dependencies: numpy:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.7 from "C:\Users\MyName\anaconda3\envs\PowerBI\python.exe" * The NumPy version is: "1.20.1"
and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help.
Original error was: DLL load failed: The specified module could not be found.
</pi>"
Anyone any ideas? I already tried lots of things myself, like:
installing a clean Anaconda virtual environment with the required libraries
remove/install numpy and setuptools as described here
Unfortunately nothing works, I keep getting the same error. Does anyone know what to do? Can it be that the Python and numpy version are not compatible? If yes, how can I find out what versions I should use that are compatible (I tried figuring this out but could not find any information)?
The NumPy version is: "1.20.1"
Currently the service supports NumPy 1.18.4.
Make sure all your modules, and version of python match the docs located here: https://learn.microsoft.com/en-us/power-bi/connect-data/service-python-packages-support
I did extensive research in the past few days and figured it out:
Anaconda is not a good option to use in combination with Power BI because you need to be able to open Power Bi from the command prompt (which I can't), see this acticle
Instead of using Anaconda I installed Python 3.8.7 locally (64-bit version) and installed the required packages
To avoid any conflicts between Python versions I deleted all other local Python installations (I did not delete Anaconda's Python versions)
Then I opened Power BI from the start menu, cleared the cache in Power BI options & settings - Data Load (see image) and then closed Power BI again.
Then I opened Power BI again from the start menu and ran my Python script... it worked!
I'm using Anaconda. I created an environment called ENGINEERING. In than environment I installed python 3.6, pandas 1.1.3, spyder 3.3.6, numpy 1.19.2, and many more. The base environment has these packages also but not necessarily the same version. Within the ENGINEERING env I created a python script in Spyder that runs without any issues when I run it in spyder. Then, I want to automate that script by creating a .bat file and then using that to automate via windows task scheduler. My .bat looks like this:
"C:\Users\alopo000\Anaconda3\envs\engineering\python.exe" "C:\Users\alopo000\Files\Python\Valid\Portf.py"
pause
when I run the .bat I get an error message:
ImportError: Unable to import required dependencies:
numpy:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.
We have compiled some common reasons and troubleshooting tips at:
https://numpy.org/devdocs/user/troubleshooting-importerror.html
Please note and check the following:
* The Python version is: Python3.6 from "C:\Users\alopo000\Anaconda3\envs\engineering\python.exe"
* The NumPy version is: "1.19.2"
and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.
Original error was: DLL load failed: The specified module could not be found.
I've tried many things to fix this. I went to the website they suggested and couldn't find anything there. I uninstalled pandas and numpy and reinstall them again but still, same error. The one thing I haven't tried is to fix the VARIABLE PATHS, mostly because I've read in many places not to change those. I can change them if that's what it is, but I would like to understand first why it is not recommended. I read it can create issues with other versions of python but not sure if that's what it is.
Do I need to activate the environment in the .bat file? is the problem that the .bat file is trying to run it outside the environment? Any help would be appreciated.
You will need to upgrade/change your base Python & numpy version installations to match those specified (3.6, 1.19.2). I had the same issue and same situation as OP (write/dev program in a virtual Spyder environment "spyder-env", then automate .py file with WTS). I tried copying over & running the 'activate.bat' in the /scripts/ folder of the virtual environment but that didn't work. In my situation, my conda spyder-env was Python = 3.9.12 and numpy = 1.21.5.
I ended up upgrading my base global Python version (conda install) for Python which changed from 3.8.3 and did the same with numpy (had to downgrade). I'm not sure if this is exactly necessary, why this works, or if it may cause problems down the line (I was about to just re-install conda).
The appropriate .bat file should be something like this once you do that:
call "C:\Users\alopo000\Anaconda3\Scripts\activate.bat"
call "C:\Users\alopo000\Files\Python\Valid\Portf.py" %*
REM Uncomment below to debug
REM pause
I want to call Python modules or user-defined functions/scripts I built in python through Matlab. I'm aware of many of the things I need to cover before something like this is possible, I'll make a list of everything I've done so far.
Created Virtual Environment with Anaconda and istalled all the dependencies I needed. Python version 3.5.2
Ran the scripts using PyCharm and all runs good/as expected.
When running pyversion(mypath) in Matlab, though it appears "loaded: 1", I can't call anything on the site-packages module. For example, I can't even call something like numpy (py.importlib.import_module('numpy')) because I get the following error:
Python Error: ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try git clean -xdf (removes all
files not under version control). Otherwise reinstall numpy.
Original error was: DLL load failed: The specified module could not be found.
I've verified everything I could so far. Python and Matlab (2016a) are both 64 bits. Although, I can run this small snippet of code with the expected outcome:
T = 'MATLAB(R) is a high-level language'
wrapped = py.textwrap.wrap(T);
whos wrapped
Since I can execute everything properly from PyCharm, and I can't from Matlab's side, I'm wondering if there's any restriction to enable from Matlab or Anaconda (in the configuration).
Is there a setting in the configuration of Anaconda that is maybe not allowing me to do this? Or is this an issue within Matlab? I'm not sure, I'm clueless.
I need to install and use the Python NumPy module (and then later the Pandas module) in order to process heavy data in Python.
I downloaded and installed ENTHOUGHT, but it wasn't what I wanted all that extra clutter of extra modules (which defeats the purpose of importing Python modules only as needed), but the uninstall did not work properly (i.e. it left garbage folders and ENTHOUGHT remnants all over my computer).
I have tried installing NumPy via EASY_INSTALL and PIP (two package managers if I understand correctly) - but with no success. Every time I try to run my program, I get the error: "no module named numpy".
I have searched the questions here and have tried to alter my ENVIRONMENT VARIABLE as per the following video, but again, no success:
https://www.youtube.com/watch?v=ddpYVA-7wq4
C:\Python34
...still the same error!
I downloaded Anaconda (with all its extra clutter and installed, but I don't like the development environment - I want my Vanilla Python IDLE to run Vanilla NumPy with no extra clutter modules...) and when I tried to again install Numpy I received a message that it was already installed with a path to:
C:\users\yoni\anaconda3\lib\site-packages
....so I ALSO added this PYTHONPATH to the ENVIRONMENT VARIABLE in hopes that it would now recognize where the NumPy installation was (currently with Anaconda3 - but I hoped to be able to import NumPy to my vanilla Python IDLE):
C:\Python34;C:\users\yoni\anaconda3\lib\site-packages
I don't find a clear answer - I see others have the same problem, and nothing is working for me. How can I finish this installation of NumPy so that it works for me when I do a simple import of module?
This is a temporary solution until you can resolve your path issue.
It will be environment specific.
import sys
sys.path.append('C:\users\yoni\anaconda3\lib\site-packages[PackageName]')
import PakcageName
I'm installing a Python (2.7.8) environment locally in my home folder on a server, in order to use newer versions of Python and some of its libraries (Numpy, Theano) together with OpenBLAS. For this, I first installed Python in the folder usr in my home directory (/home/my_home/usr/) as described here:
https://mail.python.org/pipermail/tutor/2002-March/012903.html
The installation seems to have gone smoothly. I checked this by starting the Python interpreter, which displays the right version on startup.
Following this, I installed OpenBLAS into /home/my_home/opt folder as described here:
Compiling numpy with OpenBLAS integration
... which also didn't give me any errors. However, now when I try installing Numpy (according to the instructions in the above link), I get the following error:
Traceback (most recent call last):
File "setup.py", line 22, in <module>
import subprocess
File "/home/ext_sxc/usr/lib/python2.7/subprocess.py", line 427, in <module>
import select
ImportError: No module named select
I suppose the straightforward thing to do is to find the package for select, compile and install it. I'm not sure if this is the right thing to do because the last time I installed Python 2.7.6 locally in a home directory, there were no such errors. As I understand, the select module should have been installed along with Python.
This also happens when I try installing IPython in a similar way in my home folder. The library which gives the ImportError is different.
I'm wondering if these are related to a more general mistake I may have made while installing Python. I'd appreciate any advice on this issue. Please let me know if any additional information would help. Thanks!