I have been looking for a way to solve this.
I have a python project, and this is the folder structure I want:
/project/main.py
/project/src/models.py
/project/test/tests.py
I want to be able to run the tests by executing the tests.py in terminal. tests.py imports modules in /project/src/ for testing. First I solved this by adding
sys.path.insert(0, '..') in tests.py. But then the paths used in models.py for opening text files had to be relative to the tests.py, etc. Which means the program wouldn't run when excecuted from main.py, cause of the paths.
I also tried with dots when importing modules into tests.py, like from ..src.models import *, but that gave error message saying "Attempted relative import in non-package".
What should I put in the top of tests.py to be able to import the modules from models.py?
The structure you're using is not one I would recommend, but I'm a comparaitive newb to how Python projects are usually structured. I believe this will do what you're after:
1) Place an __init__.py file inside /project, /project/src, and /project/test to make sure they're treated as packages.
2) Place from __future__ import absolute_import at the top of each Python file.
3) Then use relative imports:
test.py:
from ..src import models
main.py:
from .src import models
4) You'll need to start your application differently. Ensure your current directory is the parent of /project (which appears to be the file system root) and run your project this way:
python -m project.main
For my own project, I would definitely put main.py inside src if it's the start point of your application. I might put tests.py in src, too, but if not, I would add /project/src to the test runner's Python path on the command line instead of in code.
I would still use absolute_import regardless. In my experience, it's a very clean solution to module organization, and it's also how things work by default in Python 3.
first put your main.py in the src directory..
in your tests you can do , sys.path.append('the src directory')
if you like to force the execution to be in specific directory regardless from where you are executing the app i suggest you adding
import os
os.chdir('relative path to the src dir')
thisway your program will run in the directory you specified so it will respect the relative paths you have in your code
Related
I apologize for the millionth post about this topic.
I thought I had a good grip of the whole absolute/relative import mechanism - I even replied to a couple of questions about it myself - but I'm having a problem with it and I can't figure out how to solve it.
I'm using Python 3.8.0, this is my directory structure:
project_folder
scripts/
main.py
models/
__init__.py
subfolder00/
subfolder01/
some_script.py --> contains def for some_function
I need to import some_function from some_script.py when running main.py, so I tried:
1) relative import
# in main.py
from ..models.subfolder00.subfolder01.somescript import some_function
but when I run (from the scripts/ folder)
python main.py
this fails with error:
ImportError: attempted relative import with no known parent package
This was expected, because I'm running main.py directly as a script, so its _name_ is set to _main_ and relative imports are bound to fail.
However, I was expecting it to work when running (always from within the scripts folder):
python -m main
but I'm getting always the same error.
2) absolute import
I tried changing the import in main.py to:
# in main.py
from models.subfolder00.subfolder01.somescript import some_function
and running, this time from the main project folder:
python scripts/main.py
so that - I was assuming - the starting point for the absolute import would be the project folder itself, from which it could get to models/....
But now I'm getting the error:
ModuleNotFoundError: No module named 'models'
Why didn't it work when using the -m option in the case of relative import, and it's not working when using absolute ones either? Which is the correct way to do this?
I think quite likely you missed python's official doc ( that even come offline )
https://docs.python.org/3/tutorial/modules.html
you'll need a dummy __init__.py within your module, at same level of some_script.py
I think your "absolute" import may not have been absolute in the truest sense.
Prior to running the python scripts/main.py command, you would have needed to setup PYTHONPATH environment variable to include the path to project_folder.
Alternatively I do something like this in main.py:
import sys
import os
sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)),'..','models','subfolder00','subfolder01'))
from somescript import some_function
Maybe it is a little pedantic, but it makes sense to me.
I am working on some python project (2.7) and I have issue with imports. When I run main.py it start scripts from tests folder etc. and save output to logs and everything works fine.
/root------------
-logs
-staticCfg
-config.py
-tests
-systemTest
-scrypt1.py
-scrypt2.py
-userTest
-uScrypt1.py
main.py
My static variables (email, name etc.) are located in config.py. I need to import config.py in scrypt1.py or scrypt2.py. I tryed adding __init__.py to tests, systemTest and staticCfg folder but I always get an error.
In my scrypt1.py:
import staticCfg as cfg
...
or
from staticCfg import *
...
I get the error:
ImportError: No module named staticCfg
The import mechanism of Python can be a bit tricky.
You can refer to the documentation for more information: Python Import Mechanism
When you use absolute imports (your import does not start with a .) as you do, the import path will start from your main script (the one you launch). In your case, it's scrypt1.py. So starting from this location, python can't find the package staticCfg.
For me, the simplest solution is to create a main script in your root directory and call scrypt1.py from there (imported using from tests.systemTet import scrypt1.py). In this case, the base package will be your root folder and you will have access to the package staticCfg from all your script files as you wanted to do.
you may add root folder to PYTHONPATH.
I have the following directory structure:
app/
bin/
script1.py
script2.py
lib/
module1/
__init__.py
module1a.py
module1b.py
__init__.py
module2.py
Dockerfile
My problem is that I want to execute script1.py and script2.py, but inside those scripts, I want to import the modules in lib/.
I run my scripts from the root app/ directory (i.e. adjacent to Dockerfile) by simply executing python bin/script1.py. When I import modules into my scripts using from lib.module1 import module1a, I get ImportError: No module named lib.module1. When I try to import using relative imports, such as from ..lib.module1 import module1a, I get ValueError: Attempted relative import in non-package.
When I simply fire up the interpreter and run import lib.module1 or something, I have no issues.
How can I get this to work?
In general, you need __init__.py under app and bin, then you can do a relative import, but that expects a package
If you would structure your python code as python package (egg/wheel) then you could also define an entry point, that would become your /bin/ file post install.
here is an example of a package - https://python-packaging.readthedocs.io/en/latest/minimal.html
and this blog explains entry points quite well - https://chriswarrick.com/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/
if so, that way you could just do python setup.py install on your package and then have those entry points available within your PATH, as part of that you would start to structure your code in a way that would not create import issues.
You can add to the Python path at runtime in script1.py:
import sys
sys.path.insert(0, '/path/to/your/app/')
import lib.module1.module1a
you have to add current dir to python path.
use export in terminal
or sys.path.insert in your python script both are ok
I am struggling to successfully import my project to the test-suite in my project, as well as being able to run the program from the command-line. I've been able to run my test-suite for some time, under the impression that if the tests work, so does the command-line stuff--evidently this isn't the case. I do not yet intend on using my program as a library. The api.py acts is the entry-point for the program.
I have a project with the following structure (the same directory hierarchy as requests):
myapp/
myapp/
__init__.py
api.py # depends on commands.py
commands.py # depends on utils.py
utils.py
tests/
context.py
test_api.py # depends on api.py
test_commands.py # depends on commands.py, utils.py
In the file context.py I have a path modification adding myapp to the PYTHONPATH, so I can successfully run the tests on my code. Here is the contents of that file
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
import myapp
I've tried imaginable import combination I can think of. Far too many to list! I have also perused the Python reference import system page, and this tutorial.
How should I import my dependencies?
Turns out this was the correct layout, I mistook the error for something else. Although for future reference, relative imports in Python 3 must be explicit: when in the myapp package directory you can't say import commands, you must instead import it as from . import commands. This was defined in PEP 328 also see this SO post on the topic. Run your package with python -m mutil.api not python ./mutil/api.py as the latter won't give the interpreter context of the current path.
Suppose I have the following directory structure:
workspace/
__init__.py
ys_manage/
__init__.py
manage.py
ys_utils/
__init__.py
project_dicts.py
Now, suppose I need access to project_dicts.py in manage.py. Also, my $PATH includes /home/rico/workspace/ys_manage.
I need to be able to run manage.py from any directory on my machine and still be able to access project_dicts.py.
My $PYTHONPATH only has /home/rico/workspace.
If I include the following in manage.py I can run the file from ~/workspace/ys_manage but not anywhere else.
import sys
sys.path.append('..')
from ys_utils import project_dicts
It appears that the '..' gives a relative path to where the directory where the file is run, not where the file is located. Is this correct?
I wanted to try and use ys_manage/__init__.py to import project_dicts.py so that it would be available in manage.py universally. Is this a good idea?
I've never used __init__.py for anything other than a "package creator". That is, I've never used it for initialization purposes. Perhaps I'm doing it wrong.
Contents of ys_manage/__init__.py:
import sys
sys.path.append('..')
from ys_utils import project_dicts
Should I include something in manage.py to look for this import?
When I try and run manage.py I get the following error:
NameError: global name 'project_dicts' is not defined
As a secondary question, do I need to have workspace/__init__.py? I'd really rather not have it because ys_manage and ys_utils (and about a dozen other packages) are all under revision control and used by several other developers...workspace is not.
Generally, I've found trying to use relative paths for imports dangerous and very error prone. I'd suggest just putting workspace on your PYTHONPATH (or adding it programatically in __init__.py) and importing everything relative to that static location. It will make your code more easily readable too, as you'll be able to track down where imports are coming from much more quickly and clearly.
Try this instead of sys.path.append('..'):
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
(you'll need to import both sys and os).