I saw a lot of questions about importing modules error but I could not manage to solve the problem with Gitlab CI pipeline.
My project structure:
├───config
├───docs
├───src
__init__.py
│ ├───dataset
__init__.py
│ ├───exceptions
│ ├───resources
│ └───utils
__init__.py
└───tests
__init__.py
└───resources
__init__.py
I would like to run tests using pytest.
I invoke this command python -m pytest -p no:cacheprovider or using unittest
'python -m unittest discover -v' from root directory and also tried to invoke from test directory. In both cases I have a problem with importing class from dataset module. What's interesting, I have two tests file.
First file imports:
import os import unittest
from src.utils.FileManager import FileManager
Second imports:
from src.dataset.DatasetHelper import DatasetHelper
The first file is passing but the second is failing with error:
from dataset import DatasetHelper ModuleNotFoundError: No module
named 'dataset'
So the thing is that other modules like utils from src are imported correctly, only the dataset has a problem. I am struggling with this a few days and I completely out of ideas. I also tried to change instead of from dataset to from src.dataset. It didn't work. I can run tests in PyCharm by clicking right mous button and just run tests but not on CI environment.
What I tried:
Add modules to $PYTHONPATH like
sys.path.insert(0, "/builds/USER/PROJECT/src/dataset")
sys.path.insert(0, "/builds/USER/PROJECT/src")
sys.path.insert(0, "/builds//USER/PROJECT/tests")
The content of PYTHONPATH before adding it was:
Current $PYTHONPATH: ['/builds/USER/PROJECT/config', '/usr/local/lib/python36.zip', '/usr/local/lib/python3.6', '/usr/local/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/site-packages']
The first module in list is config because I run script from this directory to add above modules to path. Doesn't Help
Run test command from root directory and add prefix src to imports in tests directory. Doesn't Help
from dataset import DatasetHelper
ModuleNotFoundError: No module named 'dataset'
Either in src.__init__ or more proabably in src.dataset.__init__ there is the import statement from dataset import DatasetHelper. You have to rewrite it as from src.dataset import…
You can try using a relative import in your __init__.py file that's inside the tests directory.
The syntax depends on the current location as well as the current location of the module,package, or object you're trying to import. Here are some examples:
from .some_module import some_class
from ..some_package import some_function
from . import some_class
Source: https://realpython.com/absolute-vs-relative-python-imports/
Related
I have a package structured as:
Classes in those packages are named exactly like the file names. Also, init.py has following code
from tableau_util import tableau_util
from sftp_util import sftp_util
from s3_util import s3_util
I have another file e.g. test.py which is outside this folder 'utils'. I want to import those classes into test.py so my code is
from utils.tableau_util import tableau_util
from utils.sftp_util import sftp_util
from utils.s3_util import s3_util
I am still getting the error:
ModuleNotFoundError: No module named 'tableau_util'
What can I try to resolve this?
Without knowing everything I would guess that you are trying to run your test.py as a normal python script. Given this folder structure
.
├── __init__.py
├── test
│ ├── __init__.py
│ └── test.py
└── utils
├── __init__.py
├── s3_util.py
└── tableau_util.py
with these files test.py
from utils.s3_util import s3_util
from utils.tableau_util import tableau_util
s3_util()
tableau_util()
import sys
print(sys.path)
s3_util.py
def s3_util():
print('Im a s3 util!')
tableau_util.py
def tableau_util():
print('Im a tableu util!')
if you just tried to run python test/test.py in the main folder it will give you the ModuleNotFoundError. That's because it sets the ./test folder as the python path and it won't be able to see the utils folder and therefore be able to import it. However if you run it as python -m test.test (note the lack of .py you don't need it when you run it as a module) that will tell python to load it as a module and then it will run correctly with this output:
Im a s3 util!
Im a tableau util!
If you don't want to put the test.py in another folder you can simply keep it in the parent folder of utils and be able to run it in the traditional python test.py and get the same results. Error while finding spec for 'fibo.py' (<class 'AttributeError'>: 'module' object has no attribute '__path__') has some more reading on the matter.
For the record all my __init__.py files are empty and don't import anything and this is normally how they are setup unless you want to specify certain functions that need to be imported when the module is imported automatically.
I used PyCharm's create package option to create folders and files again and it is working now. Here are my new (working) folder structure:
My main script has following lines of code to import those classes:
from utils_pkg import tableau_util
from utils_pkg import s3_util
from utils_pkg import sftp_util
First, inside __init__.py (or in any sub-module that tries to import one of its siblings from the same package) you should add a "relative import" dot to the beginning of the module name, so that it reads:
from .tableau_util import tableau_util
# ^right here
Second, make sure your current working directory is not utils. A good place to start, for testing, might be to cd to the parent directory of utils instead.
My apps are organized like this:
apps/
code/
libglobal/
funglobal.py
tests/
project/
liblocal/
funlocal.py
main.py
In main.py I have:
import liblocal.funlocal
In funlocal.py I try to import funglobal.py with:
from ....code.libglobal import funglobal
When I run
python3 -B tests/project/main.py
I get an error:
from ....code.libglobal import funglobal
ValueError: attempted relative import beyond top-level package
I have read a lot of information about relative imports with python3 and still don't find how to solve this error without changing the apps organization radically. Any solution?
As the script being executed has its __name__ set as __main__ and defines itself to be on the top level of the package, it refuses to recognize scripts in sibling directories.
You can fix this with a sys.path hack:
import sys, os
sys.path.insert(0, os.path.abspath('../..'))
or an interseting alternative with setuptools is presented in this answer.
Have you a __init__.py script in each folder ?
If no, you should create an empty script named __init__.py in each folder.
I have the following directory structure:
app/
bin/
script1.py
script2.py
lib/
module1/
__init__.py
module1a.py
module1b.py
__init__.py
module2.py
Dockerfile
My problem is that I want to execute script1.py and script2.py, but inside those scripts, I want to import the modules in lib/.
I run my scripts from the root app/ directory (i.e. adjacent to Dockerfile) by simply executing python bin/script1.py. When I import modules into my scripts using from lib.module1 import module1a, I get ImportError: No module named lib.module1. When I try to import using relative imports, such as from ..lib.module1 import module1a, I get ValueError: Attempted relative import in non-package.
When I simply fire up the interpreter and run import lib.module1 or something, I have no issues.
How can I get this to work?
In general, you need __init__.py under app and bin, then you can do a relative import, but that expects a package
If you would structure your python code as python package (egg/wheel) then you could also define an entry point, that would become your /bin/ file post install.
here is an example of a package - https://python-packaging.readthedocs.io/en/latest/minimal.html
and this blog explains entry points quite well - https://chriswarrick.com/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/
if so, that way you could just do python setup.py install on your package and then have those entry points available within your PATH, as part of that you would start to structure your code in a way that would not create import issues.
You can add to the Python path at runtime in script1.py:
import sys
sys.path.insert(0, '/path/to/your/app/')
import lib.module1.module1a
you have to add current dir to python path.
use export in terminal
or sys.path.insert in your python script both are ok
I am new to python and i am having some issues with packages and imports.
My structure is as follows:
src/
base/
packcore/
__init__.py
utils/
__init__.py
util_1.py
util_2.py
helpers/
__init__.py
helper_1.py
helper_2.py
some_script.py
app.py
packcore is an external package that has been installed using pip and put into the --target=base.
some of the helpers in the packcore uses some of the utils.
From app.py i want to be able to import a helper/util.
But when i use from base.packcore.utils import some_util i get an error saying that no module named packcore from inside the helper/util
and if i do from packcore.utils import some_util i get an error no module named packcorefrom theapp.py`
help would be much appreciated :)
If you add an __init__.py to base/, you can make it a Python package to import from. You also need to make the parent a package (Which is currently called src) so it is actually a sibling module, rather than many isolated modules.
From there, you can either do an absolute import from the main package:
from src.base.packcore.helpers import helper_1
Or relative (Assuming you are in some_script.py or app.py):
from .base.packcore.helpers import helper_1
I have a parent package that has 2 child packages. It looks like this
backend
__init__.py
conf.py
db.py
connections.py
/api
__init__.py
register.py
api.py
/scheduled
__init__.py
helpers.py
All the __init__.py files are empty.
The code in backend/connections.py and backend/conf.py is being used by modules in both packages api and scheduled.
in register.py i have code like
from backend.conf import *
from backend.connections import *
Now when i do python register.py
i get this error
ImportError: No module named backend.conf
Also when i changed from backend.conf import * to from ..conf import * or from .. import conf i get this error
ValueError: Attempted relative import in non-package
What i understand by the above error is that python is not treating the above folders as packages. But i have __init__.py in all the folders. What is wrong?
When you run python register.py, your backend/register.py file is used as the __main__ module of the program, rather than as a module within the backend package. Further more, the Python import path will not automatically include the directory containing the backend directory, which is probably the cause of your problems.
One option that might work is to run your program as python -m backend.register from the top level directory of your project (or set PYTHONPATH so this module can be found). This will search for the script on the normal import path, and then run it as the main program.