We Keep Coding

Segfault when importing modules in Python virtual environment (pyenv) from console

[OS info: 5.14.14-arch1-1 x86_64 GNU/Linux]
After creating a Python v3.7.0 virtual-env with Pyenv, with the --no-site-packages option, I install packages with pip, using:
$ python -m -pip install <package-name>
An example is either Pandas v1.1.5 or Torch v1.7.0+cpu installed in the virtual-env. Then, starting a Python 3.7.0 shell in console, simply by issuing python in terminal in that environment, I try to import those packages. The result is:
$ python
Python 3.7.0 (default, Oct 20 2021, 09:16:31)
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
+>>> import faulthandler; faulthandler.enable()
+>>> import torch
Fatal Python error: Segmentation fault
Current thread 0x00007fac608f22c0 (most recent call first):
File "/home/USER/.pyenv/versions/3.7.0/lib/python3.7/ctypes/__init__.py", line 273 in _reset_cache
File "/home/USER/.pyenv/versions/3.7.0/lib/python3.7/ctypes/__init__.py", line 543 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 728 in exec_module
File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "/home/USER/.pyenv/versions/3.7.0/lib/python3.7/site-packages/torch/__init__.py", line 16 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 728 in exec_module
File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "<stdin>", line 1 in <module>
Segmentation fault
$
The interesting thing is that if in that same virtual-env, I put together an iPython kernel, like so:
$ ipython kernel install --user --name <my_project> --display-name "<Python3.7.0 (my_project)>"
and import the same package(s), say from a Jupyter notebook running that kernel, there is absolutely no issue. This tells me that there is no version issue (although dependency issues could occur still, I suppose. The segmentation fault only occurs in the Python shell in console. Also when I try importing from my pyenv global environment (3.10.0), there seems to be no issue whatsoever.
Any help to understand why and how to resolve the issue is welcome.

This is a tentative answer as I lack of a definite proof of solution. Anyone more comfortable than I am with debugging is welcome to add to it or to write their own answer.
(i) I originally thought something along the line of a corrupted ~/.inputrc might be responsible for the "hoopla" along the lines of this answer on SO. However nothing had changed in that ~/.inputrc file in past months. So the issue was more likely to be a problematic update sequence.
(ii) Looking at lines 273 and 543 in /home/USER/.pyenv/versions/3.7.0/lib/python3.7/ctypes/__init__.py the _reset_cache() function seems to trigger the segfault, just as module ctypes is imported for instance in ../torch/__init__.py.
(iii) updating pyenv to the latest version build (v2.1.0-1) and, a day later, gdbm aka GNU dbm (v1.22-1), the database function library on which Python depends, seems to have solved the issue.(Co-)incidentally both packages are maintained by the same person. ;-)
Still unsure about exactly what was at play here. Consulting the GDBM documentation did not help. Sorry about the incomplete answer.

Python env issue with Pycharm [duplicate]

This question already has answers here:
How do I use installed packages in PyCharm?
(14 answers)
Closed 1 year ago.
I'm trying to write a small script to manage my Binance account and I have an issue with a library.
I've installed the library "binance_f" from this repository https://github.com/Binance-docs/Binance_Futures_python
The library is correctly installed on default system env, and in the venv I created.
List from system env
List from venv
I can run my script from python3.8 (system or venv) console but I can't run it from pycharm.
From venv:
(venv) herve#neon-herve:~/PycharmProjects/Script-Discord-Futures$ python3.8
Python 3.8.10 (default, Jun 2 2021, 10:49:15)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import binance_f
>>> binance_f.__file__
'/home/herve/PycharmProjects/Script-Discord-Futures/venv/lib/python3.8/site-packages/binance_f/__init__.py'
>>>
From system env:
herve#neon-herve:/usr/lib/python3/dist-packages/Binance_Futures_python-master$ python3.8
Python 3.8.10 (default, Jun 2 2021, 10:49:15)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import binance_f
>>> binance_f.__file__
'/usr/lib/python3/dist-packages/Binance_Futures_python-master/binance_f/__init__.py'
I tried to use venv or system env with my PyCharm project but it failed.
Traceback (most recent call last):
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/herve/PycharmProjects/Script-Discord-Futures/binance_functions.py", line 8, in <module>
import binance_f
ModuleNotFoundError: No module named 'binance_f'
This is the interpreter in PyCharm:
System interpreter in PyCharm
Venv interpreter in PyCharm
In Pycharm, I tried to manually add the library but I have an error message. (I added the repository, and tried to install "binance_f".
Pycharm error
I really need to use PyCharm for the debug function !
Thanks a lot for your help.

So as it was clear the issue came from PyCharm I started to check all option available. I found an option in the execution configuration:
Solution
I just force PyCharm to run my script in Python Console !
Thanks to all for your help.

PyJulia build fails; can't find package 'Pkg' in current path during import?

I am trying to build PyJulia on my system but get the following error when trying to install via Python:
>>> import julia
>>> julia.install()
ERROR: LoadError: ArgumentError: Package Pkg not found in current path:
- Run `import Pkg; Pkg.add("Pkg")` to install the Pkg package.
Stacktrace:
[1] require(::Module, ::Symbol) at .\loading.jl:823
[2] include at .\boot.jl:326 [inlined]
[3] include_relative(::Module, ::String) at .\loading.jl:1038
[4] include(::Module, ::String) at .\sysimg.jl:29
[5] exec_options(::Base.JLOptions) at .\client.jl:267
[6] _start() at .\client.jl:436
in expression starting at C:\Users\wgorman\AppData\Local\Continuum\anaconda3\lib\site-packages\julia\install.jl:12
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\wgorman\AppData\Local\Continuum\anaconda3\lib\site-packages\julia\tools.py", line 102, in install
raise PyCallInstallError("Installing", output)
julia.tools.PyCallInstallError: Installing PyCall failed.
When I install via Julia, I make sure to point the build installation to the correct Python interpreter and libpython, however, when I try to make a simple call within Python it says that those variables haven't been defined.
>>> import julia
>>> from julia import Main
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
File "C:\Users\wgorman\AppData\Local\Continuum\anaconda3\lib\site-packages\julia\core.py", line 246, in load_module
JuliaMainModule(self, fullname))
File "C:\Users\wgorman\AppData\Local\Continuum\anaconda3\lib\site-packages\julia\core.py", line 148, in __init__
self._julia = loader.julia
File "C:\Users\wgorman\AppData\Local\Continuum\anaconda3\lib\site-packages\julia\core.py", line 238, in julia
self.__class__.julia = julia = Julia()
File "C:\Users\wgorman\AppData\Local\Continuum\anaconda3\lib\site-packages\julia\core.py", line 482, in __init__
raise UnsupportedPythonError(jlinfo)
julia.core.UnsupportedPythonError: It seems your Julia and PyJulia setup are not supported.
Julia executable:
julia
Python interpreter and libpython used by PyCall.jl:
None
None
Python interpreter used to import PyJulia and its libpython.
C:\Users\wgorman\AppData\Local\Continuum\anaconda3\python.exe
C:\Users\wgorman\AppData\Local\Continuum\anaconda3\python37.dll
julia> versioninfo()
Julia Version 1.0.5
Commit 3af96bcefc (2019-09-09 19:06 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i7-2600 CPU # 3.40GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.0 (ORCJIT, sandybridge)
Python 3.7.3 (default, Mar 27 2019, 17:13:21) [MSC v.1915 64 bit (AMD64)]
Windows-10-10.0.18362-SP0
8 logical CPU cores, Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
Issue posted on github here: https://github.com/JuliaPy/pyjulia/issues/382

I'm guessing PyJulia can't find your desired Julia environment. I too have had difficulties with getting PyJulia to work in the past. This was what worked for me:
pip install julia
Then in a python file or Jupyter notebook:
# Import Julia
from julia.api import Julia
jpath = "julia-1.3.1/bin/julia" # path to Julia, from current directory (your path may be slightly different)
jl = Julia(runtime=jpath, compiled_modules=False) # compiled_modules=True may work for you; it didn't for me
# Import Julia Modules
from julia import Main
Main.include("path/to/some_julia_file.jl")
jl.eval("using .some_julia_module") # use a module from some_julia_file.jl
# Evaluate Stuff
this_equals_three = jl.eval("1+2")
# eval works with Python f-strings, so you can easily use python variables
a = 1
b = 2
also_three = jl.eval(f"{a}+{b}")
result = jl.eval(f"myfunc({a},{b})") # call custom functions imported from some_julia_file.jl
from julia import Base
equals_one = Base.sind(90)

No Module Named '_pywrap_tensorflow_internal'

While trying to validate the installation of tensorflow-gpu, I get an ImportError when trying to execute "import tensorflow as tf". I am using a Quadro K620 on Windows 7. Tensorflow was installed using pip.
The following is the stack trace:
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.
C:\Users\aagarwal>python
Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AM
D64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
Traceback (most recent call last):
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\site-packag
es\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_hel
per
return importlib.import_module(mname)
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\importlib\_
_init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
File "<frozen importlib._bootstrap>", line 577, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 906, in create_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
ImportError: DLL load failed: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\site-packag
es\tensorflow\python\pywrap_tensorflow.py", line 41, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\site-packag
es\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\site-packag
es\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_hel
per
return importlib.import_module('_pywrap_tensorflow_internal')
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\importlib\_
_init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: No module named '_pywrap_tensorflow_internal'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\site-packag
es\tensorflow\__init__.py", line 24, in <module>
from tensorflow.python import *
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\site-packag
es\tensorflow\python\__init__.py", line 51, in <module>
from tensorflow.python import pywrap_tensorflow
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\site-packag
es\tensorflow\python\pywrap_tensorflow.py", line 52, in <module>
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\site-packag
es\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in swig_import_hel
per
return importlib.import_module(mname)
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\importlib\_
_init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 986, in _gcd_import
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
File "<frozen importlib._bootstrap>", line 577, in module_from_spec
File "<frozen importlib._bootstrap_external>", line 906, in create_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
ImportError: DLL load failed: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\site-packag
es\tensorflow\python\pywrap_tensorflow.py", line 41, in <module>
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\site-packag
es\tensorflow\python\pywrap_tensorflow_internal.py", line 21, in <module>
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\site-packag
es\tensorflow\python\pywrap_tensorflow_internal.py", line 20, in swig_import_hel
per
return importlib.import_module('_pywrap_tensorflow_internal')
File "C:\Users\aagarwal\AppData\Local\Programs\Python\Python35\lib\importlib\_
_init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: No module named '_pywrap_tensorflow_internal'
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/install_sources#common_installation_probl
ems
for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.
>>>
I have looked at multiple other stack overflow posts which things like correcting the path but I have not been able to solve this issue.

I came across the same issue today, please switch to cuDNN v5.1 Library for Windows instead as #mickdelaney suggested and then try to
Check environment settings of CUDA, normally all the settings of CUDA had been added to Windows environment
Copy files in bin, lib and include of cuDNN to bin, lib and include of CUDA respectively. Normally the directory is C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA
And then you can import tensorflow and run your code. Good luck!

I found the solution for the CPU: It worked for me
Just run below command if necessary then update it
pip install tensorflow==1.5

you could uninstall the current version of tensorflow and try a low version such as TensorFlow 0.12 using the code
pip install --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-0.12.0rc0-cp35-cp35m-win_amd64.whl
https://blog.csdn.net/baidu_36669549/article/details/79734590

After you install the CUDA Drivers you must download cuDNN and release it. After that you get to copy the cudnn64_5.dll to PATH.
If that doesn't work, move the cudnn64_5.dll to
NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin and try it again. Otherwise you'd meet some error that you see.

https://github.com/tensorflow/tensorflow/issues/7705
switch to cuDNN 5.1 instead of 6.0

I had the same error with anaconda, it was previously installed with another program through anaconda. Installing again through conda-forge worked: conda install -c conda-forge tensorflow

I still got this error after installing cuDNN 5.1 (yes, I had PATH set correctly). I had better luck installing tensorflow_gpu from this site. This requires cuDNN 6.0, not 5.1. Unfortunately they currently only have Tensorflow 1.1 for GPU systems.

this is all about your video chip/card and CPU compatibility with TF VERSION! Current version of Tensor Flow is 1.8 and it crashes on both my machines.
I have a win7 Gateway notebook using an I5 CPU and intel video chip. I have a desktop win7 unit with AMD Phenom II and AMD Radeon HD5450 video card. The I5 notebook requires V1.6 of TensorFlow and the desktop AMD unit takes V1.5. I don't have a GPU available for running ANN's so I don't know how that would affect it.
Here's the command line for PIP install: pip install --upgrade tensorflow==1.5.0 (or whatever other version you need). This will uninstall your current version and then install the indicated version.
Build or copy a small TensorFlow script and run it in a command window. Open a second command window and install a different version. I'd recommend starting at 1.5 and working your way up.

The only thing that worked for me (Windows 10) is:
Install Microsoft Visual C++ 2015 Redistributable Update 3
If you have a new fresh computer setup this one might miss.

I am a chinese, so my english is poor, and my answer is that however you use cudnn 7 or 6 or 5 , you must rename the dll from cudnn64_x.dll to cudnn64_6.dll after you copy files.
maybe soon, you need to rename the dll to 7 or 8 or 9 as possible.

It worked for me after doing this command :
pip install tensorflow-gpu == 1.15.5
I was previously with tensorflow-gpu 1.15.0 when it was showing the error

Problems installing "MATLAB Engine for Python" with Anaconda

I am using Python 3.4.3 | Anaconda 2.3.0 (x86_64) on OS X 10.10.5 and following these instructions to install MATLAB Engine for Python. Installation seems to succeed, but upon launching python & attempting to import, I see the following:
Python 3.4.3 |Anaconda 2.3.0 (x86_64)| (default, Oct 20 2015, 14:27:51)
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import matlab.engine
Traceback (most recent call last):
File "/Users/tyler/.local/lib/python3.4/site-packages/matlab/engine/__init__.py", line 42, in <module>
pythonengine = importlib.import_module("matlabengineforpython"+_PYTHONVERSION)
File "/Users/tyler/anaconda/lib/python3.4/importlib/__init__.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2224, in _find_and_load_unlocked
ImportError: No module named 'matlabengineforpython3_4'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/tyler/.local/lib/python3.4/site-packages/matlab/engine/__init__.py", line 58, in <module>
pythonengine = importlib.import_module("matlabengineforpython"+_PYTHONVERSION)
File "/Users/tyler/anaconda/lib/python3.4/importlib/__init__.py", line 109, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
File "<frozen importlib._bootstrap>", line 539, in _check_name_wrapper
File "<frozen importlib._bootstrap>", line 1715, in load_module
File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
ImportError: dlopen(/Applications/MATLAB_R2015b.app/extern/engines/python/dist/matlab/engine/maci64/matlabengineforpython3_4.so, 2): Library not loaded: #rpath/libpython3.4m.dylib
Referenced from: /Applications/MATLAB_R2015b .app/extern/engines/python/dist/matlab/engine/maci64/matlabengineforpython3_4.so
Reason: image not found
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/tyler/.local/lib/python3.4/site-packages/matlab/engine/__init__.py", line 60, in <module>
raise EnvironmentError('The installation of MATLAB Engine for Python is '
OSError: The installation of MATLAB Engine for Python is corrupted. Please reinstall it or contact MathWorks Technical Support for assistance.
While researching this issue, I found several other references to this exact issue on MathWorks' website, all without a resolution. The final link suggests the problem may be with dynamic linking.
Given how many scientific toolboxes are only available in MATLAB, it would be of great use to be able to call this engine using Anaconda. While there are other packages available that support this functionality, they invariably have limitations or other complexity like a client-server architecture.
Any suggestions are most welcome.

According to my communication with MathWorks support, MATLAB Engine for Python is incompatible with Anaconda. Fortunately, this is incorrect.
Thanks to some direction from the Anaconda mailing list, I found a simple solution:
sudo ln -s ~/anaconda/lib/libpython3.4m.dylib /usr/lib
After this, matlab.engine imports successfully. Hope this helps someone!
Edit: As of OS X 10.11 El Capitan, the workaround needs to be tweaked due to the Rootless feature, as root no longer has permission to modify /usr/lib
sudo ln -s ~/anaconda/lib/libpython3.4m.dylib /usr/local/lib

If you're trying to install matlab engine in a conda environment, follow the below steps:
cd "matlabroot\extern\engines\python"
python setup.py install --prefix="installdir"
Eg
cd /opt/MATLAB/R2019a/extern/engines/python
python setup.py install --prefix="/home/nagabhushan/anaconda3/"
Now, matlab engine package is actually installed under the separate
anaconda3/lib/python3.6/site-package/matlab
So you need to manually move that matlab folder to the site-package folder your python is using. If you have multiple virtual envs, move it to the corresponding site-package folder of the virtual env you will run program.
Eg
anaconda3/envs/my_env/lob/python3.6/site-packages/matlab
References:
https://stackoverflow.com/a/41800724/3337089
https://stackoverflow.com/a/56553740/3337089
Edit 1 (03-Mar-2020):
I tried activating my environment and then simply installing matlab engine and it worked! I'm using Python 3.7
python setup.py install
Note that this initially gave error that error: You do not have write permission in build/lib/matlab/engine/, but changing the permission of build directory recursively worked: sudo chmod -R 777 build/