We Keep Coding

Absolute/relative import in Python: ModuleNotFoundError and more

This is my project structure:
- config
- data
- src
- resources
- db
- test
N.B.: I am using Python 3.9 and every folder that contains a .py file also has a __init__.py file
All the scripts I want to run are located in the /src folder and they used code from other scripts placed in the /src/resources folder (which is basically acting like a library).
Some of these scripts also read YAML files from the /config folder
Here is the problem, I cannot find a way to properly run these scripts from the command line, I am always getting errors like:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/runpy.py", line 185, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/local/lib/python3.8/runpy.py", line 111, in _get_module_details
__import__(pkg_name)
File "/home/pi/crypto/src/ethMessage.py", line 4, in <module>
import update_db
File "/home/pi/crypto/src/update_db.py", line 1, in <module>
from db.mysql_main import insertValueAndFee
File "/home/pi/crypto/src/db/mysql_main.py", line 6, in <module>
from src.resources.parser import read_yaml
ModuleNotFoundError: No module named 'src'
I tried both with relative and absolute import, right now absolute import is what I am using (e.g. from src.resources.parser import read_yaml)
Which is the proper way to run scripts from the command line?
EDIT:
As you suggested, I added
sys.path.append( os.path.abspath(os.path.dirname(__file__)+'/..') )
to all the main scripts, and I am still getting a similar error:
Traceback (most recent call last):
File "src/ethMessage.py", line 6, in <module>
import update_db
File "/home/pi/crypto/src/update_db.py", line 1, in <module>
from db.mysql_main import insertValueAndFee
File "/home/pi/crypto/src/db/mysql_main.py", line 6, in <module>
from src.resources.parser import read_yaml
ModuleNotFoundError: No module named 'src'
To clarify, I am running my script from the global folder, which in my case is named "crypto".
I am also open to change the project structure with one that doesn't create problems.

If you want to refer to all of those packages by their root name, then all you have to do is add that folder to the Python path. So, for main program scripts in src, just add something like this:
import os
import sys
sys.path.append( os.path.abspath(os.path.dirname(__file__)+'/..') )
Now, the parent directory of your script will be on the path, no matter where you run it from. Now you can say
from src.resources.parser import read_yaml

If someone is still looking for a solution, I highly recommend not to bother with Python's imports: they are probably the worst part of the whole language.
Instead, if you want to use some files as a library, you should use setuptools to create a package from those files.
Then, you can install it locally or publish it on PyPi.
This way, you can import your library in a script just like another third-party module, (e.g. requests, selenium, ...), and things will work, instead of giving you a headache because a file is in a directory instead of another.

Exception during calling gensim?

I tried to load gensim in my code. Often it works fine. Today, I get the following exception:
Traceback (most recent call last):
File "/project/6008168/tamouze/just.py", line 2, in <module>
import gensim
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/gensim/__init__.py", line 5, in <module>
from gensim import parsing, corpora, matutils, interfaces, models, similarities, summarization, utils # noqa:F401
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/gensim/parsing/__init__.py", line 4, in <module>
from .preprocessing import (remove_stopwords, strip_punctuation, strip_punctuation2, # noqa:F401
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/gensim/parsing/preprocessing.py", line 40, in <module>
from gensim import utils
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/gensim/utils.py", line 44, in <module>
from smart_open import smart_open
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/smart_open/__init__.py", line 1, in <module>
from .smart_open_lib import *
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/smart_open/smart_open_lib.py", line 29, in <module>
import requests
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/requests/__init__.py", line 97, in <module>
from . import utils
File "/project/6008168/tamouze/Python_directory/ENV2.7_new/lib/python2.7/site-packages/requests/utils.py", line 26, in <module>
from ._internal_utils import to_native_string
ImportError: cannot import name to_native_string
Im using python 2.7.14 and gensim 3.4.0.
How can I solve this problem?

The error isn't really occurring in gensim, even though that's how you found it. If you look at the stack, it's only triggered because gensim uses smart_open which in turn uses requests. It is in requests that the error happens.
If this was working, but now stopped, something likely changed in your environment, or how you're launching this code, related to the relationship between Python and the requests package.
For such errors, you should try searching Google for the final-two lines of your error stack – those most connected to the problem. Those are:
from ._internal_utils import to_native_string
ImportError: cannot import name to_native_string
(These leave out the file path that's specific to you, but have a number of unique tokens likely to have also been reported by any others.)
A number of people have hit this, from a variety of other projects, but always triggered through requests. Some have reported re-installing requests (perhaps to ensure it's version 2.0.0 or later) helps.
If a simple re-install doesn't help, you could also try one or all of:
uninstall, verify it's not present at all (that requests itself isn't found), then install – this could make sure you don't have overlapping redundant installations in different places that are both confusing the issue
start from a fresh Python environment, reinstalling all packages
double-check that all packages share the same Python2/Python3 compatibility

No module named 'json' after installing simplejson

I am working in Ubuntu 14.04 and I have multiple versions of Python on my machine (they include python2.7 and python3.4). Few days back, I installed simplejson on my system. I don't remember how I did that but I guess it must be similar to pip install simplejson. However, now a strange problem has started appearing when I try installing any python package. For example, just now I tried installing Tkinter using sudo pip3.4 install Tkinter and it throws the following error:
Traceback (most recent call last):
File "/usr/local/bin/pip3.4", line 9, in <module>
load_entry_point('pip==1.5.4', 'console_scripts', 'pip3.4')()
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 351, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2363, in load_entry_point
return ep.load()
File "/usr/lib/python3/dist-packages/pkg_resources.py", line 2088, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
File "/usr/lib/python3/dist-packages/pip/__init__.py", line 61, in <module>
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "/usr/lib/python3/dist-packages/pip/vcs/subversion.py", line 4, in <module>
from pip.index import Link
File "/usr/lib/python3/dist-packages/pip/index.py", line 15, in <module>
from pip.wheel import Wheel, wheel_ext
File "/usr/lib/python3/dist-packages/pip/wheel.py", line 25, in <module>
from distlib.scripts import ScriptMaker
File "/usr/share/python-wheels/distlib-0.1.8-py2.py3-none-any.whl/distlib/scripts.py", line 15, in <module>
File "/usr/share/python-wheels/distlib-0.1.8-py2.py3-none-any.whl/distlib/resources.py", line 20, in <module>
File "/usr/share/python-wheels/distlib-0.1.8-py2.py3-none-any.whl/distlib/util.py", line 11, in <module>
ImportError: No module named 'json'
Sometimes I can fix this if the error tells me that in one of the files I have:
import json
which I simply convert to
import simplejson as json
I tried uninstalling simplejson:
sudo pip uninstall simplejson
but it gives me the same error: json not found.
Can anybody please help me fix this so that I would happily be able to install python packages? Thanks in advance.

Note: I do not have a definitive answer but will offer a series of steps you can try:
The first thing is see if you can import json from the usual python interpreter:
import json
print(json.__file__) #this would be important to know if it works
If that does work (as well as commenting what json.__file__ is) you would then want to try to use pip from the interpreter.
If you can't import json normally:
This is not surprising, I did not expect pip to be looking in a non-standard place for modules. You will want to figure out where the json package should be located on your computer, you can do this by importing another module from the standard library and looking at it's __file__:
>>> import fractions
>>> fractions.__file__
'/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/fractions.py'
This will obviously be different for you but I'd expect there to be a json folder in the same folder as fractions.py
if you can't import fractions or queue or datetime etc.
If you can't import anything from the standard library you will probably want to just reinstall python.
If the json folder is there and contains an __init__.py
Use the rename function of your file browser to make sure there are no weird special characters, but other then that I'm not sure, if you can import fractions.py but not a package from the same folder that would imply there is something very wrong with the import mechanics of your python version.
If the json folder is not with the rest of the standard library
It is possible that your python distribution has a different structure then I'd expect, it at least can't hurt to take a look for it.
You can search for the json folder amongst your various python files using the find command, not really sure how it works but just another thing to try. If you do find it with the __init__.py, encode.py, decode.py, scanner.py, and tool.py (at least those are the ones in my version) you'll probably want to figure out how it got there, but maybe just move it to the same folder as the rest of the standard library.
If you can't find the json package or you find it and it is corrupted
Well then you will need to replace it! Don't worry, this isn't too hard, just grab a source release of python from the site and extract the json package from it, once it is uncompressed the json folder should be in the Lib folder. Simply copy/move it to the rest of the standard library and you should be good to go!
I hope this helps you debug what is going on, This covers all the scenarios I could imagine happening and I would be interested in which one fixed your issue (or what you were able to figure out so I can come up with more options)

I am guessing that you install it using either pip install simplejson to download from PyPI or using apt-get install python-simplejson to download from ubuntu repositories.
It is possible that you downlaoded the library for the Python2 if you used any of the commands above and it won't be available for Python3 (which pip3.4 will use). Can you try these commands and help debug yourself?
$ python -c "import simplejson"
$ python3.4 -c "import simplejson"
This would tell you which version of the python did you install simplejson for last time (my guess in python2). If the 2nd command errors out with ImportError try:
$ pip3.4 install simplejson
and then install your libraries.

Error Importing SHTOOLS Python Package

I am trying to import the spherical harmonic toolbox (SHTOOLS) in python. I have the files downloaded and unzipped and am using RedHat.
I added the package's path to my python system path and when I go to import the package, I get this error:
>>import pyshtools
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyshtools/__init__.py", line 49, in <module>
load_documentation()
File "pyshtools/__init__.py", line 27, in load_documentation
from . import _SHTOOLS
ImportError: cannot import name _SHTOOLS
I can not seem to figure out what the issue is. I checked that the path to this folder was actually added to the system path and it was.
Is this an issue on my end? Or is it possible that I have something downloaded incorrectly? If so, how would I go about fixing this issue?

The SHTOOLS package needs to be built with make first to compile the Fortran libraries. The wiki on Github gives directions on which libraries are required - libblas-dev, liblapack-dev, g++, gfortran, and libfftw3-dev (these are the Ubuntu packages, they may have slightly different names on Redhat). Once these are installed, you need to run make, then sudo make all to install the Fortran and Python components. The Makefile has a lot of good comments in it, I'd recommend reading through it before running make.

importing module in Spyder error

I am trying to use sklearn in Spyder. At the beginning when I tried to import it I was gettingImportError: No module named sklearn
Then I tied to set the PATH with PYTHONPATH manager and then use 'Update module names list' from tools menu.then restart the spydet but no success.
at the end I copied the sklearn folder to /Applications/Spyder.app/Contents/Resources/lib/python2.7
This is how I find the PATH of sklearn and copied in into this folder :
>>> import sklearn
>>> sklearn
<module 'sklearn' from '/Library/Python/2.7/site-packages/scikit_learn-0.12_git-py2.7-macosx-10.7-intel.egg/sklearn/__init__.pyc'>
>>>
and then I cd to the parent folder :
cp -r sklearn /Applications/Spyder.app/Contents/Resources/lib/python2.7
but when I try to import the sklearn to spyder like from sklearn import dataset :
Traceback (most recent call last):
File "/Users/mohsenjadidi/Documents/workspace/dsv/test.py", line 10, in <module>
from sklearn import dataset
File "/Applications/Spyder.app/Contents/Resources/lib/python2.7/sklearn/__init__.py", line 17, in <module>
from .base import clone
File "/Applications/Spyder.app/Contents/Resources/lib/python2.7/sklearn/base.py", line 11, in <module>
from .metrics import r2_score
File "/Applications/Spyder.app/Contents/Resources/lib/python2.7/sklearn/metrics/__init__.py", line 6, in <module>
from .metrics import confusion_matrix, roc_curve, auc, precision_score, \
File "/Applications/Spyder.app/Contents/Resources/lib/python2.7/sklearn/metrics/metrics.py", line 17, in <module>
from ..utils import check_arrays
File "/Applications/Spyder.app/Contents/Resources/lib/python2.7/sklearn/utils/__init__.py", line 9, in <module>
from .murmurhash import murmurhash3_32
File "numpy.pxd", line 151, in init sklearn.utils.murmurhash (sklearn/utils/murmurhash.c:4773)
ValueError: numpy.dtype has the wrong size, try recompiling
any idea?Thanks

Moj, Spyder MacOS X App uses its own (internal) Python interpreter, not the system one. This was the only way we (the devs) found to provide a self contained app with numpy, scipy, matplotlib and IPython, without messing with system Python.
Right now I think there is no way to add more packages to the app, but we'll try to improve the situation during the next couple of months.
The other alternative (the one we envisaged) is for users who want more packages to download and install latest EPD and change their Python interpreter in
Tools > Preferences > Console > Advanced Settings > Python Executable

I faced the same problem importing networkx. Drag-drop its folder into spyder's folder was enough for me.