So I'm running into issues when I run pytest versus when I invoke the tested module directly. Pytest actually runs fine but I run into issues trying to directly invoke the python file that I'm testing. Here is the basic folder layout:
generate
├── generate
│ ├── __init__.py
│ ├── write_sql.py
│ └── utils.py
└── tests
└── test_write_sql.py
Within both test_write_sql.py and write_sql.py I'm importing things using the generate module like so:
from generate import utils
Running pytest from the root generate folder works fine. But when I try to invoke write_sql.py directly from the root generate folder I run into the following error:
root#637572f508b9:~/generate# python3 generate/write_sql.py
Traceback (most recent call last):
File "/root/generate/generate/write_sql.py", line 5, in <module>
from generate import utils
ModuleNotFoundError: No module named 'generate'
My impression was that running Python from the generate root folder should add the generate subfolder as a module to the system path (due to the included __init__.py file). I know if I change the import to import utils inside write_sql.py then the direct call to write_sql.py works but then pytest runs into import errors.
Any ideas as to what import scheme I should use to get both pytest and the direct invocation working?
I was having the same issues with imports, the way i fixed it was setting all the imports so that the code runs. In order to run the tests instead of just calling pytest (this runs all tests from the folder where they are located, therefore there will be import error) it should be called from the root directory like this:
python3 -m pytest
or
python -m pytest
this way it runs all tests from the root directory, and it should not give import errors
Related
I'm trying to build a project via the method described in The Hitchhiker's Guide To Python. I'm running into problems with the test structure.
My file structure looks like this:
.
├── __init__.py
├── sample
│ ├── __init__.py
│ └── core.py
├── setup.py
└── tests
├── __init__.py
├── context.py
└── test_core.py
With:
# sample/core.py
class SampleClass:
def got_it(self):
return True
And:
# tests/context.py
import os
import sys
sys.path.insert(0, os.path.abspath(
os.path.join(os.path.dirname(__file__), '..')
))
import sample
And:
# tests/test_core.py
import unittest
from .context import sample
class SampleClassTest(unittest.TestCase):
def test_got_it(self):
# ...
pass
if __name__ == '__main__':
unittest.main()
(Note that I just thru the __init__.py files in the root and tests to see if that helped, but it didn't.)
When I try to run tests/test_core.py with Python 3.7. I get this error:
ImportError: attempted relative import with no known parent package
That happens if I run the test file from outside the tests directory or in it.
If I remote the . and do this in tests/test_core.py:
from context import sample
Everything loads, but I can't access SampleClass.
The things I've tried are:
sc = SampleClass()
NameError: name 'SampleClass' is not defined
sc = sample.SampleClass()
AttributeError: module 'sample' has no attribute 'SampleClass'
sc = sample.core.SampleClass()
AttributeError: module 'sample' has no attribute 'core'
sc = core.SampleClass()
NameError: name 'core' is not defined
I tried on Python 2 as well and had similar problems (with slightly different error methods). I also tried calling a function instead of a class and had similar problems there as well.
Can someone point me in the direction of what I'm doing wrong?
This once caused me some headaches, until I realized that it just depended on the way the test is started. If you use python tests/test_core.py, then you are starting a simple script outside any package. So relative imports are forbidden.
But if (still from the project root folder) you use:
python -m tests.test_core
then you are starting the module test_core from the tests package, and relative imports are allowed.
From that time, I always start my tests as modules and not as scripts. What is even better is that unittest discover knows about packages. So if you are consistent in naming the test folder and scripts with a initial test, you can forget about the number of tests and just use:
python -m unittest discover
and that is enough to run the whole test suite.
Happy testing!
Since you have a proper package complete with setup.py, you're better off using pip install -e . instead of using a sys.path hack.
Minimal setup.py
#setup.py
from setuptools import setup, find_packages
setup(name='sample', version='0.0.1', packages=find_packages(exclude=('tests')))
Then you should import everything from sample packages anywhere you want using the same syntax (i.e. absolute path). To import SampleClass for example:
from sample.core import SampleClass
Also, to answer your question on why you're getting on relative import:
ImportError: attempted relative import with no known parent package
You'd have to avoid invoking the file as a script using python ... because relative imports do not work on scripts. You should invoke the file as a module using python -m .... Here is a very nice explanation that's been viewed a billion times:
I have a package structured as:
Classes in those packages are named exactly like the file names. Also, init.py has following code
from tableau_util import tableau_util
from sftp_util import sftp_util
from s3_util import s3_util
I have another file e.g. test.py which is outside this folder 'utils'. I want to import those classes into test.py so my code is
from utils.tableau_util import tableau_util
from utils.sftp_util import sftp_util
from utils.s3_util import s3_util
I am still getting the error:
ModuleNotFoundError: No module named 'tableau_util'
What can I try to resolve this?
Without knowing everything I would guess that you are trying to run your test.py as a normal python script. Given this folder structure
.
├── __init__.py
├── test
│ ├── __init__.py
│ └── test.py
└── utils
├── __init__.py
├── s3_util.py
└── tableau_util.py
with these files test.py
from utils.s3_util import s3_util
from utils.tableau_util import tableau_util
s3_util()
tableau_util()
import sys
print(sys.path)
s3_util.py
def s3_util():
print('Im a s3 util!')
tableau_util.py
def tableau_util():
print('Im a tableu util!')
if you just tried to run python test/test.py in the main folder it will give you the ModuleNotFoundError. That's because it sets the ./test folder as the python path and it won't be able to see the utils folder and therefore be able to import it. However if you run it as python -m test.test (note the lack of .py you don't need it when you run it as a module) that will tell python to load it as a module and then it will run correctly with this output:
Im a s3 util!
Im a tableau util!
If you don't want to put the test.py in another folder you can simply keep it in the parent folder of utils and be able to run it in the traditional python test.py and get the same results. Error while finding spec for 'fibo.py' (<class 'AttributeError'>: 'module' object has no attribute '__path__') has some more reading on the matter.
For the record all my __init__.py files are empty and don't import anything and this is normally how they are setup unless you want to specify certain functions that need to be imported when the module is imported automatically.
I used PyCharm's create package option to create folders and files again and it is working now. Here are my new (working) folder structure:
My main script has following lines of code to import those classes:
from utils_pkg import tableau_util
from utils_pkg import s3_util
from utils_pkg import sftp_util
First, inside __init__.py (or in any sub-module that tries to import one of its siblings from the same package) you should add a "relative import" dot to the beginning of the module name, so that it reads:
from .tableau_util import tableau_util
# ^right here
Second, make sure your current working directory is not utils. A good place to start, for testing, might be to cd to the parent directory of utils instead.
I have a directory structure:
root_dir
├── src
│ └── p1.py
└── lib
├── __init__.py
├── util1.py
└── util2.py
I want to run src/p1.py which uses lib/util1.py using an import statement import lib.util1 as u1.
It runs fine when I use PyCharm, but I want to also run it from command line. How can I run the program from command line?
I have tried cd root_dir then python src/p1.py.
But it produces the following error:
Traceback (most recent call last):
File "./src/p1.py", line 1, in <module>
import lib.util1 as u1
ImportError: No module named lib.util1
How can I run the python program src/p1.py from the command line?
Edit: Based on the suggestion from #Sumedh Junghare, in comments, I have added __init__.py in lib folder. But still it produces the same error!
You need the following steps
Add __init__.py at lib folder.
Add this line at p1.py file on top
import sys
sys.path.append('../')
import lib.util1 as u1
Run the p1.py file from src dir. Hope it will work.
Edit:
If you do not want to add sys.path.append('../'), set PYTHONPATH in env-var from this resource.
How to add to the pythonpath in Windows?
Improving on Saiful's answer, You can do the following which will allow you to run the your program from any working directory
import sys
import os
sys.path.append(os.path.join(os.path.realpath(os.path.dirname(__file__)), "../"))
import lib.util1 as u1
I have read a ton of stackoverflow answers and a bunch of tutorials. In addition, I tried to read the Python documentation, but I cannot make this import work.
This is how the directory looks like:
myDirectory
├── __init__.py
├── LICENSE
├── project.py
├── README.md
├── stageManager.py
└── tests
├── __init__.py
└── test_project.py
There is a class in project.py called Project, and I want to import it in a file under tests directory. I have tried the following:
Relative import:
from ..project import Project
def print_sth():
print("something")
This gives me the following error: (running from the tests directory as python test_project.py and from myDirectory as python tests/test_project.py)
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from ..project import Project
SystemError: Parent module '' not loaded, cannot perform relative import
Absolute import with package name:
If I have something like the following, I get ImportError (with the same run command as above).
from project import Project
def print_sth():
print("something")
------------------------------------------------------
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from project import Project
ImportError: No module named 'project'
and this too:
from myDirectory.project import Project
def print_sth():
print("something")
------------------------------------------------------
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from myDirectory.project import Project
ImportError: No module named 'myDirectory'
Finally, I tried adding the if __name__ == '__main__' statement within the test_project.py file, but it still failed. I would really appreciate if anyone could help. If there is a solution where I do not have to write a verbose command, I would prefer that.
When you run a Python script by filename, the Python interpreter assumes that it is a top-level module (and it adds the directory the script is in to the module search path). If the script is in a package, that's not correct. Instead, you should run the module using the -m flag, which takes a module name in the same format as an import statement (dotted separators) and puts the current directory in the module search path.
So, you could run the test from myDirectory with: python -m tests.test_project. When you run the script this way, either of the kinds of imports you tried will work.
But if myDirectory is supposed to be a top-level package itself (as the __init__.py file suggests), you should probably go up one level further up, to myDirectory's parent, and run the script with two levels of package names: python -m myDirectory.tests.test_project. If you do this and want the test to use an absolute import you'd need to name the top level package that the project module is in: from myDirectory.project import Project.
I'm building a set of Python modules that depend on each other.
My file directory currently looks like:
.
├── utilities
│ ├── __init__.py
│ ├── utility.py
│ ├── data.csv
├── src
│ ├── __init__.py
│ |── functions
│ ├── __init__.py
│ └── function.py
└── __init__.py
Further, function.py imports data from utilities/data.csv.
From the top-level directory (.), I run python3 src/functions/function.py.
And I receive the following import error:
Traceback (most recent call last):
File "src/functions/function.py", line 1, in <module>
from utilities.utility import UtilityFunctionA
ImportError: No module named 'utilities'
How do I properly import utilities from the function.py file? Or should I not be running nested files in the first place, and instead running files at the top-level of the directory?
The imports are successful when running this code from within PyCharm.
Silly question, but I've been unable to figure it out despite reading a lot of documentation (and Googling).
UPDATE:
Using python3 -m src.functions.function works to run the module from the command line with proper imports and with successfully loading the csv.
However, when I then run the module from within PyCharm, for instance using
Now I receive the error that OSError: File b'utilities/data.csv' does not exist
Is there any way to setup my module to run both from within PyCharm and also from the command line?
If you want to be able to do - from utilities.utility import UtilityFunctionA - from within function.py , when running from the top level directory - . . You need to run the python file as a module , using -m option. Example -
python3 -m src.functions.function
But The above method only works if you always run from the . directory (top-level directory) . I always either add the top-level directory manually into the PYTHONPATH environment variable.
Or use os.path and relative path to add it to sys.path programmatically using __file__. Example -
import sys
import os.path
path_to_top = os.path.abspath(os.path.join(os.path.dirname(__file__),'..','..'))
sys.path.append(path_to_top)
After this do the import. If the directory structures would always remain the same, this would work even in other systems, without having to set any environment variable.
As per the updated requirements -
Now I receive the error that OSError: File b'utilities/data.csv' does not exist
Is there any way to setup my module to run both from within PyCharm and also from the command line?
For reading files, Python uses the current working directory as the start point , and all relative paths are resolved relative to the current working directory. It is never a good idea to rely on the current working directory to be a particular directory as we can run python scripts from anywhere using absolute path to the script. For loading files as well, we can use os.path and relative paths to create the absolute path to the file. Example -
import os.path
path_to_datacsv = os.path.abspath(os.path.join(os.path.dirname(__file__),'..','..','utilities,'data.csv'))
This imports properly:
python3 -m src.functions.function
Based on How to do relative imports in Python?