I have a project I want to run on different machines without the need to modify the PYTHONPATH enviroment variable. My projects structure is as follows:
awesome_project/
data/
scripts/
__init__.py
predict/
importer/
__init__.py
__init__.py
predict.py
train/
importer/
__init__.py
__init__.py
train.py
utils/
__init__.py
configuration.py
In my predict and train code I need to import variables defined in the configuration file inside utils. In Python 2 I defined the importer module, in which the __init__.py had the following code:
import sys
from os import getcwd
from os.path import sep
root_path = sep.join(getcwd().split(sep)[:-2])
sys.path.append(root_path)
And it worked as a charm. I imported the variables as: from scripts.utils.configuration import models_path, but now I'm migrating my code to Python 3 and this does not work at all, I get the following error:
Traceback (most recent call last):
File "predict.py", line 11, in <module>
from scripts.utils.configuration import models_path
ModuleNotFoundError: No module named 'scripts.utils'
What am I doing wrong?
Related
I have a directory structure:
root_dir
├── src
│ └── p1.py
└── lib
├── __init__.py
├── util1.py
└── util2.py
I want to run src/p1.py which uses lib/util1.py using an import statement import lib.util1 as u1.
It runs fine when I use PyCharm, but I want to also run it from command line. How can I run the program from command line?
I have tried cd root_dir then python src/p1.py.
But it produces the following error:
Traceback (most recent call last):
File "./src/p1.py", line 1, in <module>
import lib.util1 as u1
ImportError: No module named lib.util1
How can I run the python program src/p1.py from the command line?
Edit: Based on the suggestion from #Sumedh Junghare, in comments, I have added __init__.py in lib folder. But still it produces the same error!
You need the following steps
Add __init__.py at lib folder.
Add this line at p1.py file on top
import sys
sys.path.append('../')
import lib.util1 as u1
Run the p1.py file from src dir. Hope it will work.
Edit:
If you do not want to add sys.path.append('../'), set PYTHONPATH in env-var from this resource.
How to add to the pythonpath in Windows?
Improving on Saiful's answer, You can do the following which will allow you to run the your program from any working directory
import sys
import os
sys.path.append(os.path.join(os.path.realpath(os.path.dirname(__file__)), "../"))
import lib.util1 as u1
I have read a ton of stackoverflow answers and a bunch of tutorials. In addition, I tried to read the Python documentation, but I cannot make this import work.
This is how the directory looks like:
myDirectory
├── __init__.py
├── LICENSE
├── project.py
├── README.md
├── stageManager.py
└── tests
├── __init__.py
└── test_project.py
There is a class in project.py called Project, and I want to import it in a file under tests directory. I have tried the following:
Relative import:
from ..project import Project
def print_sth():
print("something")
This gives me the following error: (running from the tests directory as python test_project.py and from myDirectory as python tests/test_project.py)
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from ..project import Project
SystemError: Parent module '' not loaded, cannot perform relative import
Absolute import with package name:
If I have something like the following, I get ImportError (with the same run command as above).
from project import Project
def print_sth():
print("something")
------------------------------------------------------
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from project import Project
ImportError: No module named 'project'
and this too:
from myDirectory.project import Project
def print_sth():
print("something")
------------------------------------------------------
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from myDirectory.project import Project
ImportError: No module named 'myDirectory'
Finally, I tried adding the if __name__ == '__main__' statement within the test_project.py file, but it still failed. I would really appreciate if anyone could help. If there is a solution where I do not have to write a verbose command, I would prefer that.
When you run a Python script by filename, the Python interpreter assumes that it is a top-level module (and it adds the directory the script is in to the module search path). If the script is in a package, that's not correct. Instead, you should run the module using the -m flag, which takes a module name in the same format as an import statement (dotted separators) and puts the current directory in the module search path.
So, you could run the test from myDirectory with: python -m tests.test_project. When you run the script this way, either of the kinds of imports you tried will work.
But if myDirectory is supposed to be a top-level package itself (as the __init__.py file suggests), you should probably go up one level further up, to myDirectory's parent, and run the script with two levels of package names: python -m myDirectory.tests.test_project. If you do this and want the test to use an absolute import you'd need to name the top level package that the project module is in: from myDirectory.project import Project.
I have the following directory structure:
Chippa/
bin/
__init__.py
app.py
tests/
__init__.py
app_tests.py
tools.py
templates/
hello_form.html
index.html
layout.html
docs/
In my app_tests.py file, I have:
from nose.tools import *
from bin.app import app
from tests.tools import assert_response
When I try to run app_tests.py from outside of the tests directory, one level above the test directory inside the Chippa directory, like so:
python tests/app_tests.py
I get the following error:
Traceback (most recent call last):
File "tests/app_tests.py", line 3, in <module>
from bin.app import app
ImportError: No module named bin.app
But I do have an empty __init__.py in the bin directory, which I thought would have prevented this issue. What am I missing here?
For that import to work, you have to actually be running a proper module in the first place. So, rather than
python tests/app_tests.py
, try
python -m tests.app_tests
No matter how I structure the imports in the code files and in the __init__.py files, I can't seem to get it right for executing the program and running the tests using pytest. How do I need to write the imports when my project structure looks like this:
src/
__init__.py
VocableFileWriter.py
WordListHelper.py
WordListReader.py
XLDAttributeValueAdder.py
exceptions/
__init__.py
XMLInvalidException.py
XMLParseException.py
gui/
__init__.py
GTKSignal.py
XLDAttributeValueAdderWindow.py
test/
__init__.py
test_XLDAttributeValueAdder.py
xmlparser/
__init__.py
XMLParser.py
Currently I have them like this:
In the __init__.py files I have the imports like this (src/__init__.py):
from src import *
from src.exceptions import *
from src.xmlparser import *
and in a subpackage (src/xmlparser/__init__.py):
from src.xmlparser import *
So I guess those are project-absolute paths to the modules.
In the code files themselves I import like this:
import os
import sys
from VocableFileWriter import VocableFileWriter
from XLDAttributeValueAdder import XLDAttributeValueAdder
However, when I execute the code from the directory above src using:
./src/main.py
It tells me that:
Traceback (most recent call last):
File "./src/main.py", line 6, in <module>
from VocableFileWriter import VocableFileWriter
File "/home/xiaolong/Development/PycharmProjects/xld-attribute-value-adder/src/VocableFileWriter.py", line 2, in <module>
from XMLInvalidException import XMLInvalidException
ImportError: No module named 'XMLInvalidException'
It used to be a PyCharm project, but I couldn't get it to run with the imports structure PyCharm used when not using PyCharm but running it from the terminal, so I decided I wanted to take the whole import stuff into my own hands. So don't get confused by it being in a PyCharm directory.
I also want to be able to execute the tests using for example:
py.test src/test/test_XLDAttributeValueAdder.py
How do I solve this mess?
Edit#1:
I had the program running once, but then the test complained about missing modules, probably because it's in another subdirectory and I tried so many configurations, that I don't know how I had the program running anymore. If possible please add some explanation why a certain structure is correct and works for both, tests and the program itself.
EDIT#2:
I've managed to get the tests running now, but now the program doesn't run anymore.
I emptied all the __init__.py files and used only project-absolute paths in the code files like this (src/test/test_XLDAttributeValueAdder.py):
from src.VocableFileWriter import VocableFileWriter
from src.WordListHelper import WordListHelper
from src.WordListReader import WordListReader
from src.XLDAttributeValueAdder import XLDAttributeValueAdder
from src.exceptions.XMLInvalidException import XMLInvalidException
from src.exceptions.XMLParseException import XMLParserException
from src.xmlparser.XMLParser import XMLParser
from src.test.path_helper import go_up
from src.test.path_helper import go_in
and in the main.py:
from src.VocableFileWriter import VocableFileWriter
from src.XLDAttributeValueAdder import XLDAttributeValueAdder
the output of ./src/main.py:
./src/main.py
Traceback (most recent call last):
File "./src/main.py", line 6, in <module>
from src.VocableFileWriter import VocableFileWriter
ImportError: No module named 'src'
EDIT#3:
I've tried the relative import way like this:
main.py moved one directory up
The structure now looks like this:
main.py
src/
__init__.py
VocableFileWriter.py
WordListHelper.py
WordListReader.py
XLDAttributeValueAdder.py
exceptions/
__init__.py
XMLInvalidException.py
XMLParseException.py
gui/
__init__.py
GTKSignal.py
XLDAttributeValueAdderWindow.py
test/
__init__.py
test_XLDAttributeValueAdder.py
xmlparser/
__init__.py
XMLParser.py
__init__.py files empty
main.py file gets relative
Imports look like this:
import os
import sys
from .src.VocableFileWriter import VocableFileWriter
from .src.XLDAttributeValueAdder import XLDAttributeValueAdder
When I try to run the main.py file:
Traceback (most recent call last):
File "main.py", line 6, in <module>
from .src.VocableFileWriter import VocableFileWriter
SystemError: Parent module '' not loaded, cannot perform relative import
However, there is a __init__.py file in the same directory as the main.py file, also empty.
You'd better use relative imports.
When doing from src import * in src/__init__.py you'll only import anything you've defined before calling the import. Most likely that's just nothing. If you've defined the __all__ variable, you'll get the submodule from it, but maybe also an AttributeError if a variable hasn't been defined yet.
So instead import the modules you need explicitly where you need them like
from .VocableFileWriter import VocableFileWriter
from .exceptions.XMLInvalidException import XMLInvalidException
or lets say within src/gui/GTKSignal.py
from ..exceptions.XMLInvalidException import XMLInvalidException
Also you could use project-absolute paths like mentioned in your Edit#2.
Furthermore you've a problem with your path when calling ./src/main.py. In this case the directory containing src is not in the path. Instead ./src/ is in the path. Therefore python doesn't consider it a package anymore and since there's no other package with that name available, the import fails.
Instead put your main.py module in the same directory like the src package or call it as a module with python -m src.main. Doing so you could also rename it to src/main.py and call the package instad with python -m src.
When using the first approach, main should use an absolute import structure. Just think of it as a module you put anywhere on your computer, while your src package is somewhere else, where it can be found by Python, like any other module or package installed by the system. It's not different, when it's located besides the package itself, since the current directory is in sys.path.
main.py
import os
import sys
from src.VocableFileWriter import VocableFileWriter
from src.XLDAttributeValueAdder import XLDAttributeValueAdder
I have created a Flask application in portal/webapp.py. When I try to start the application using python portal/webapp.py I get the following error:
Traceback (most recent call last):
File "portal/webapp.py", line 3, in <module>
from db import TenantManager, QueryHandler
File "****/Project/portal/db.py", line 4, in <module>
from sql_parser.SQLParserTools import Parser, Builder
ImportError: No module named sql_parser.SQLParserTools
Here is my project structure
Project/
portal/
__init__.py
db.py
manage.py
test/
__init__.py
test_db.py
webapp.py
sql_parser/
__init__.py
error.py
SQLParserTools.py
StringParsers.py
test/
__init__.py
test_parser.py
I think you should add ..../Project/ to your PYTHONPATH variable.
Note that since there is no __init__.py in Project. You consider that portal and sql_parser are two different project. One which is using the other.