Image of my file and Folder layout
Trying to import modules from grandparent files. Searching told me to use init.py files. I actually fixed it using them, and my file structure was:
main.py
app.py
...
Tests |
| - __init__.py
| - testrunner.py
| ...
In this example, testrunner would import a module from app.py.
This worked, but recently added more testfiles and wanted to organize more. So now my file structure is:
main.py
app.py
...
Tests |
| - __init__.py
| - Sprint 1
| - __init__.py
| - testrunner.py
| ...
Now it doesn't work anymore. I can move them back into Tests and the file runs fine again, but not in Sprint 1. There I still get the "E ModuleNotFoundError: No module named 'App'" error
I tried deleting and recreating the __init__.py files, reopening the project, adding (import sys
sys.path.append('..')) to testrunner.py. None of it worked, though I can't for the life of me figure out why adding the __init__.py files didn't work now for some reason.
Related
A have several DAGs of similar structure and I wanted to use advice described in Airflow docs as Modules Management:
This is an example structure that you might have in your dags folder:
<DIRECTORY ON PYTHONPATH>
| .airflowignore -- only needed in ``dags`` folder, see below
| -- my_company
| __init__.py
| common_package
| | __init__.py
| | common_module.py
| | subpackage
| | __init__.py
| | subpackaged_util_module.py
|
| my_custom_dags
| __init__.py
| my_dag1.py
| my_dag2.py
| base_dag.py
In the case above, these are the ways you could import the python
files:
from my_company.common_package.common_module import SomeClass
from my_company.common_package.subpackage.subpackaged_util_module import AnotherClass
from my_company.my_custom_dags.base_dag import BaseDag
That works fine in Airflow.
However I used to validate my DAGs locally by running (also as advised by a piece of documentation - DAG Loader Test):
python my_company/my_custom_dags/my_dag1.py
When using the imports, it complains:
Traceback (most recent call last):
File "/[...]/my_company/my_custom_dags/my_dag1.py", line 1, in <module>
from my_company.common_package.common_module import SomeClass
ModuleNotFoundError: No module named 'my_company'
How should I run it so that it understands the context and reckognizes the package?
It works when run this way:
PYTHONPATH=. python my_company/my_custom_dags/my_dag1.py
It seems that when entry point is my_dag1.py that's inside my_company/my_custom_dags Python considers it as its "working directory" and only looks for modules within that path. When I add . to PYTHONPATH it can also look at entire directory structure and reckognize module my_company and below.
(I'm not expert in Python, so above explanation might be somewhat innacurrate, but this is my understanding. I'm also not sure if that's indeed the cleanest way to make it work)
I moved my tests to a separate subfolder in the project, and now testing my classes does not work anymore.
|-- project
| |main.py
| |-- lib
| | |__init__.py
| | |myclass.py
| |-- tests
| | |__init__.py
| | |test_myclass.py
Both init files are empty.
but when I run the test (i'm in the tests folder and typing python -m unittest) I get the following error:
ModuleNotFoundError: No module named 'lib'
at the line from lib.myclass import Myclass
I also saw that one could use sys and os to add the parent folder to the path, but each time that I do it, the code that adds it is automatically going under all the imports in vscode (I guess due to some automatic formatting?) and therefore is not ran on time.
That's because once you are in tests folder your working directory dir is:
.\
.\test_mycalss.py
You should run tests from project so your working directory will cover whole tree:
.\
.\main.py
.\lib\
and so on
Your project cannot view lib.myclass because in folder tests there is no folder lib.
I have a python 2.7 project which I have structured as below:
project
|
|____src
| |
| |__pkg
| |
| |__ __init__.py
|
|____test
|
|__test_pkg
| |
| |__ __init__.py
|
|__helpers
| |
| |__ __init__.py
|
|__ __init__.py
I am setting the src folder to the PYTHONPATH, so importing works nicely in the packages inside src. I am using eclipse, pylint inside eclipse and nosetests in eclipse as well as via bash and in a make file (for project). So I have to satisfy lets say every stakeholder!
The problem is importing some code from the helpers package in test. Weirdly enough, my test is also a python package with __init__.py containing some top level setUp and tearDown method for all tests. So when I try this:
import helpers
from helpers.blaaa import Blaaa
in some module inside test_pkg, all my stakeholders are not satisfied. I get the ImportError: No module named ... and pylint also complains about not finding it. I can live with pylint complaining in test folders but nosetests is also dying if I run it in the project directory and test directory. I would prefer not to do relative imports with dot (.).
The problem is that you can not escape the current directory by importing from ..helpers.
But if you start your test code inside the test directory with
python3 -m test_pkg.foo
the current directory will be the test directory and importing helpers will work. On the minus side that means you have to import from . inside test_pkg.
I would understand this behaviour.
In Pycharm I can successfully run (green play button) this main.py which includes src.v0.Det.py
src
|
+--v0
| |
| +--Det.py (class det)
| __init__.py
|
+-- main.py
__init__.py
main.py
------------
from src.v0.Det import det
....
But when I run on unix using: python main.py I get "No module named src.v0.Det".
I can obviously remove "src" and it works but I don't want to change the file. How can I run this file like in Pycharm?
Riccardo
export PYTHONPATH="${PYTHONPATH}:<.....path.to.src.without.src....>"
I'm trying to set Sphinx autodoc for my Flask project.
Here is schema of my project:
Folder
|--docs
| |--build
| |--source
| conf.py
| index.rst
| mod_ololo.rst
| ....
|--flask_app
| |--celery_worker.py
| |--config.py
| |--run.py
| |--app
| |--__init__.py
| |--temp.py
| |--mod_search
| ....
| |--mod_files
| |--__init__.py
| ....
and so on.
In conf.py I already set:
sys.path.insert(0, os.path.abspath("../.."))
I'm trying to add content of temp.py to my documentation.
Here is content of mod_ololo.rst:
*******************************************
OLOLO module
*******************************************
Ololololo
======
.. automodule:: flask_app.app.temp
:members:
And when I run make html for separate scripts - Sphinx autodoc works, but in case of my flask project it doesn't and shows me the following error:
ImportError: No module named 'config'
Where config is config.py
Purely a guess, but the config module that your sphinx builder isn't finding is probably the config.py module in your flask_app folder. There's likely an import config statement somewhere in your flask_app/app/temp.py file, so when sphinx tries to run the automodule procedure on it, it gets a failure on that import.
From your directory structure it looks like your flask_app folder is not a python package, so you might have to add that folder rather than (or in addition to) its parent folder to the path in your conf.py file:
sys.path.insert(0, os.path.abspath('../../flask_app'))
Also, you'll likely have to change your automodule statement in mod_ololo.rst not to reference flask_app as a package:
.. automodule:: app.temp
If you actually do intend to use flask_app as a package, then you need an __init__.py file in that folder, and your import statements within your project should be absolute from the top flask_app package, i.e.:
from flask_app import config
rather than
import config