We Keep Coding

relative import with no known parent package error while importing from different folder

I've a directory structure like this below:
ParentPackage/
childPackage/
__init__.py
subpackageOne/
__init__.py
moduleA.py
moduleB.py
test/
__init__.py
test.py
I'm trying to import a method "methodX" from moduleA.py to my test.py residing inside test folder.
below is the code which I've tried
import os
import sys
sys.path.append(os.path.realpath('..'))
from ..subpackageOne.moduleAimport methodX
but getting error "ImportError: attempted relative import with no known parent package".
In addition to this moduleA.py is importing moduleB.py and while running the test I am getting No module named 'moduleB'.
There are already lot of question has been asked on this topic, however suggestions provided for those didn't work for me.

Based on your architecture, you can do that:
import os
import sys
# Assuming your package is on a drive called C:.
# That is what I mean by absolute path, it needs to refer the path from the disk
# so it does not depend on where you are
sys.path.append("C:/path/to/childPackage")
# Absolute import is just importing wihtout using dots at the beginning.
# In this case, python will look for installed packages in your environment
# and then any folder that is in your path.
from subpackageOne.moduleA import methodX
If you prefer, you may want to add the parentPackage in your path so you can use any childPackage in case you have several.
sys.path.append("C:/path/to/parentPackage")
from childPackage.subpackageOne.moduleA import methodX
Note: I see you have __init__ files in your architecture which suggests you may want to create an installable package that you can build after (using a setup.py or setup.cfg file). In this case, you could install your package in your virtual environment and then you would not need to add the directory to your path. setuptools Doc or some tutorial

Why do I get an import error when a child file is trying to access a parent file

My directory is structured like this
>project
>tests
>test_to_json.py
>app.py
>x.db
in my test_to_json.py I want to access a class from app.py
How would I import it? I've tried from ..app import myClass but that gives me ImportError: attempted relative import with no known parent package. Is there something I'm missing?

You cannot use .. relative pathing in python. That specific type of relative python is simply not allowed. . is allowed, though. The resolution for this problem is usually done by converting your project into a python package.
Extensive tutorials for doing so can be found here, but I will give an example of how to convert your project into a package.
Step 1
The new file structure should look like this:
>project
>tests
>__init__.py #Note this file
>test_to_json.py
>__init__.py #Note this file
>setup.py #Note this file
>app.py
>x.db
Step 2
Write your setup.py.
Here is an generic setup.py that should work for your project:
from setuptools import setup, find_packages
setup(
name='project_package', #the name of your desired import
version='0.0.1',
author='An Awesome Coder',
packages=find_packages(),
description='An awesome package that does something',
install_requires=[], # a list of python dependencies for your package
)
find_packages() looks for all the __init__.py files in your package to identify submodules.
Step 3
Install your package. In the folder with your new setup.py, run pip install -e . This will install your package on your computer, basically adding the files to your python system path.
Step 4
From any python terminal on your computer you should now be able to import your package using the package_name you specified.
import project_package
from project_package.app import myClass
myClass()
.
.
.

Why is relative path not working in python tests?

My directory layout is as follows
project\
project\setup.py
project\scripts\foo.py
project\scripts\bar.py
project\scripts\__init__.py
project\tests\test_foo.py
project\tests\__init__.py
My test file looks as follows
project\tests\test_fo.py
from ..scripts import foo
def test_one():
assert 0
I get the following error, when I do
cd C:\project
C:\virtualenvs\test_env\Scripts\activate
python setup.py install
python setup.py test
E ValueError: Attempted relative import beyond toplevel package
What am I doing wrong?
This is my setup.py
setup(
name = 'project',
setup_requires=['pytest-runner'],
tests_require=['pytest'],
packages = ["scripts","tests"],
package_data={
'scripts': ['*.py'],
'tests': ['*.py'],
},
)

Relative imports only work within a package. scripts may be a package, and so is tests, but the project directory is not (and neither should it be). This makes scripts and tests top-level packages. You can't refer to other top-level names using relative syntax.
Moreover, tests are not run with the tests package; the test runner imports the test_foo module, not the tests.test_foo module, so as far as Python is concerned test_foo is a top-level module.
scripts is a top-level name, just use that directly. You will have to add the project directory to sys.path however. You can do so at the top of your test_foo.py file with:
import os
import sys
TEST_DIR = os.path.dirname(os.path.abspath(__file__))
PROJECT_DIR = os.path.abspath(os.path.join(TEST_DIR, os.pardir))
sys.path.insert(0, PROJECT_DIR)
then import from scripts with absolute paths:
from scripts import foo
Note however that when you run python setup.py then your current working directory is added to sys.path anyway, so scripts is available directly without having to fiddle with sys.path.
Moreover, pytest will already do the work for you; for any given test file it'll make sure the first parent directory with no __init__.py file in it is on sys.path. In your case that's the project/ directory, so again scripts is directly available to import from. See Good Practices:
If pytest finds a “a/b/test_module.py” test file while recursing into the filesystem it determines the import name as follows:
determine basedir: this is the first “upward” (towards the root) directory not containing an __init__.py. If e.g. both a and b contain an __init__.py file then the parent directory of a will become the basedir.
perform sys.path.insert(0, basedir) to make the test module importable under the fully qualified import name.
import a.b.test_module where the path is determined by converting path separators / into ”.” characters. This means you must follow the convention of having directory and file names map directly to the import names.
Note that in order to actually use pytest to run your tests when you use setup.py test, you need to register an alias in your setup.cfg file (create it in project/ if you do not have one):
[aliases]
test = pytest

Python imports structure

I want to have this structure for my project:
requirements.txt
README.md
.gitignore
project/
__init__.py
project.py
core/
__init__.py
base.py
engines/
__init__.py
engine1.py
engine2.py
utils/
__init__.py
refine_data.py
whatever.py
The application is run from project/project.py. However, I constantly get import errors when using relative or absolute imports.
Both engines need to import from project.core.base, the utils need to import from project.core.base as well, and project.py (the main file ran) needs to be able to import from engines.
Absolute imports don't work:
# engines/engine1.py
from project.core.base import MyBaseClass
which gives the error:
ImportError: No module named project.core.base
But if I try a relative import instead
# engines/engine1.py
from ..core.base import MyBaseClass
I get:
ValueError: Attempted relative import beyond toplevel package
I've seen other projects on Github structured similarly, but this seems to cause all sorts of problems. How do I get this to work?

Take a look at your sys.path. It's likely that the top project directory is in the python path, and it sees your sub-packages (ie. utils, engines, etc.) as separate packages, which is why it's giving you an error that you're trying to import from outside your package when doing relative imports, and absolute imports don't work because it can't find the top project directory because it's not under any of the python paths.
The directory above the top project directory is what needs to be added to the python path.
Ex.
/path/is/here/project/core/...
# Add this to the PYTHONPATH
/path/is/here

Try to use these imports:
engine1.py:
from core import base
refine_data.py:
from core import base
project.py
from engines import engine1
if you use pycharm mark project directory as sources root and then try to run project.py. If you don't use pycharm you can run project.py by going to project directory and running command:
python project.py

How can I create imports that always work?

I am struggling a bit to set up a working structure in one of my projects. The problem is, that I have main package and a subpackage in a structure like this (I left out all unnecessary files):
code.py
mypackage/__init__.py
mypackage/work.py
mypackage/utils.py
The utils.py has some utility code that is normally only used in the mypackage package.
I normally have some test code each module file, that calls some methods of the current module and prints some things to quickcheck if everything is working correctly. This code is placed in a if __name__ == "__main__" block at the end of the file. So I include the utils.py directly via import utils. E.g mypackage/work.py looks like:
import utils
def someMethod():
pass
if __name__ == "__main__":
print(someMethod())
But now when I use this module in the parent package (e.g. code.py) and I import it like this
import mypackage.work
I get the following error:
ImportError: No module named 'utils'
After some research I found out, that this can be fixed by adding the mypackage/ folder to the PYTHONPATH environment variable, but this feels strange for me. Isn't there any other way to fix this? I have heard about relative imports, but this is mentioned in the python docs about modules
Note that relative imports are based on the name of the current module. Since the name of the main module is always "main", modules intended for use as the main module of a Python application must always use absolute imports.
Any suggestions how I can have a if __name__ == "__main__" section in the submodule and also can use this file from the parent package without messing up the imports?
EDIT: If I use a relative import in work.py as suggested in a answer to import utils:
from . import utils
I get the following error:
SystemError: Parent module '' not loaded, cannot perform relative import

Unfortunately relative imports and direct running of submodules don't mix.
Add the parent directory of mypackage to your PYTHONPATH or always cd into the parent directory when you want to run a submodule.
Then you have two possibilities:
Use absolute (from mypackage import utils) instead of relative imports (from . import utils) and run them directly as before. The drawback with that solution is that you'll always need to write the fully qualified path, making it more work to rename mypackage later, among other things.
or
Run python3 -m mypackage.utils etc. to run your submodules instead of running python3 mypackage/utils.py.
This may take some time to adapt to, but it's the more correct way (a module in a package isn't the same as a standalone script) and you can continue to use relative imports.
There are more "magical" solutions involving __package__ and sys.path but they all require extra code at the top of every file with relative imports you want to run directly. I wouldn't recommend these.

You should create a structure like this:
flammi88
├── flammi88
│ ├── __init__.py
│   ├── code.py
│   └── mypackage
│   ├── __init__.py
│   ├── utils.py
│   └── work.py
└── setup.py
then put at least this in the setup.py:
import setuptools
from distutils.core import setup
setup(
name='flammi88',
packages=['flammi88'],
)
now, from the directory containing setup.py, run
pip install -e .
This will make the flammi88 package available in development mode. Now you can say:
from flammi88.mypackage import utils
everywhere. This is the normal way to develop packages in Python, and it solves all of your relative import problems. That being said, Guido frowns upon standalone scripts in sub-packages. With this structure I would move the tests inside flammi88/tests/... (and run them with e.g. py.test), but some people like to keep the tests next to the code.
Update:
an extended setup.py that describes external requirements and creates executables of the sub-packages you want to run can look like:
import setuptools
from distutils.core import setup
setup(
name='flammi88',
packages=['flammi88'],
install_requires=[
'markdown',
],
entry_points={
'console_scripts': [
'work = flammi88.mypackage.work:someMethod',
]
}
)
now, after pip installing your package, you can just type work at the command line.

Import utils inside the work.py as follows:
import mypackage.utils
or if you want to use shorter name:
from mypackage import utils
EDIT: If you need to run work.py outside of the package, then try:
try:
from mypackage import utils
except ImportError:
import utils

Use:
from . import utils
as suggested by Peter
In your code.py you should use:
from mypackage import work