I have a python project where I use grpc.
I create the files with python -m grpc_tools.protoc -I "pathToMyProtoFile" --python_out=. --grpc_python_out=. "pathToMyProtoFile\module.proto"
I want all the grpc-stuff to be in a python package. So I created a sub folder "my_package_folder" and added an empty __init__.py in it.
My Problem: How to access and where to place the generated module_pb2.py and module_pb2_grpc.py.
If I place them into the root folder of my application I cannot access them from my package with from .. import module_pb2_grpc "attempted relative import beyond top-level package"
If I place them into my "my_package_folder" the 2 generated files do not find each other.
(import module_pb2 as module__pb2 in "module_pb2_grpc.py")
This import mechanism in python is so extremely confusing... I have no idea where to start to solve this problem.
My folder structure is just the main project folder and a sub folder "my_package_folder" for all the grpc stuff.
Let's say you have a folder structure like this. I'm just taking the example of one file.
├── module_pb2_grpc.py
├── my_package_folder
│ ├── __init__.py
Then to resolve the attempted relative import beyond top-level package, you can add this.
init.py
import os
import sys
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.append(os.path.dirname(SCRIPT_DIR))
from module_pb2_grpc import *
Related
Although I think this should be pretty simple I still can't make it run.
I have following folder structure:
├── apartment
│ ├── src
│ ├── train_model
│ ├── __init__.py
│ ├── train_model.py
│ ├── utils.py
│ ├── interference.py
│ └── __init__.py
In utils.py I tried:
from src.interference import create_sample
Error: ModuleNotFoundError: No module named 'src'
from .interference import create_sample
Error: ImportError: attempted relative import with no known parent package
from interference import create_features_sample
ModuleNotFoundError: No module named 'interference'
What is the way to make it work? I am not a huge fan of non-pythonic ways as it looks dirty.
The structure starting with a src/ is aimed explicitly at not enabling import by from src.intereference import ..., and you should not put an __init__.py file in src/ folder.
Instead, following nice explanation and examples here: https://blog.ionelmc.ro/2014/05/25/python-packaging/, here is what I recommend:
install the package:
add a setup.py file at root of your folder (this is clearly not as hard as it seems)
maybe create a virtual environment
using pip install -e . (with trailing dot!) command
then simply import your package by from interference import ...
To answer to your primary request, you could update src/__init__.py with from intereference import create_sample, to expose this function at a higher level, then the chained import would work. However, I do not recommend this, as it renders everything very rigid.
You need to add the directory that contains interference to PYTHONPATH.
You can use OS depending path in "module search path" which is listed in sys.path. So you can easily add parent directory like following:
import sys
sys.path.insert(0, '..')
from interference import create_features_sample
Note that the previous code uses a relative path, so you must launch the file inside the same location or it will likely not work.
To launch from anywhere, you can use Path from the pathlib module.
from pathlib import Path
import sys
path = str(Path(Path(__file__).parent.absolute()).parent.absolute())
sys.path.insert(0, path)
from interference import create_features_sample
If you want add a custom path to your PYTHONPATH permanently, go to the 'site-packages' folder of your current Python environment and add the file 'custompaths.pth' in which each line should consist of a directory which will then be checked if you try to import the module.
Assuming 'src' being the module you want to import you should add the following line to the .pth file:
your_preceding_path/apartment
Error because of python can't able to find the specific file or package. Python usually find only for the child packages or files, otherwise you need to specify the absolute path.
import sys, os
sys.path.append(os.path.abspath(os.path.join('../..', 'src')))
Please refer the "How to access a module from outside your file folder in Python?"
Have you tried from ..interference import create_sample ?
Or for the whole module from .. import interference.
I've checked here, that I'm using another command, something that is missing from the question at the time of writing this. The command that I'm using is python -m src.train_model.utils from apartment folder.
Thanks to RMPR for helping me.
This problem is similar to this one.
I have a package structured as:
Classes in those packages are named exactly like the file names. Also, init.py has following code
from tableau_util import tableau_util
from sftp_util import sftp_util
from s3_util import s3_util
I have another file e.g. test.py which is outside this folder 'utils'. I want to import those classes into test.py so my code is
from utils.tableau_util import tableau_util
from utils.sftp_util import sftp_util
from utils.s3_util import s3_util
I am still getting the error:
ModuleNotFoundError: No module named 'tableau_util'
What can I try to resolve this?
Without knowing everything I would guess that you are trying to run your test.py as a normal python script. Given this folder structure
.
├── __init__.py
├── test
│ ├── __init__.py
│ └── test.py
└── utils
├── __init__.py
├── s3_util.py
└── tableau_util.py
with these files test.py
from utils.s3_util import s3_util
from utils.tableau_util import tableau_util
s3_util()
tableau_util()
import sys
print(sys.path)
s3_util.py
def s3_util():
print('Im a s3 util!')
tableau_util.py
def tableau_util():
print('Im a tableu util!')
if you just tried to run python test/test.py in the main folder it will give you the ModuleNotFoundError. That's because it sets the ./test folder as the python path and it won't be able to see the utils folder and therefore be able to import it. However if you run it as python -m test.test (note the lack of .py you don't need it when you run it as a module) that will tell python to load it as a module and then it will run correctly with this output:
Im a s3 util!
Im a tableau util!
If you don't want to put the test.py in another folder you can simply keep it in the parent folder of utils and be able to run it in the traditional python test.py and get the same results. Error while finding spec for 'fibo.py' (<class 'AttributeError'>: 'module' object has no attribute '__path__') has some more reading on the matter.
For the record all my __init__.py files are empty and don't import anything and this is normally how they are setup unless you want to specify certain functions that need to be imported when the module is imported automatically.
I used PyCharm's create package option to create folders and files again and it is working now. Here are my new (working) folder structure:
My main script has following lines of code to import those classes:
from utils_pkg import tableau_util
from utils_pkg import s3_util
from utils_pkg import sftp_util
First, inside __init__.py (or in any sub-module that tries to import one of its siblings from the same package) you should add a "relative import" dot to the beginning of the module name, so that it reads:
from .tableau_util import tableau_util
# ^right here
Second, make sure your current working directory is not utils. A good place to start, for testing, might be to cd to the parent directory of utils instead.
This question already has answers here:
How to do relative imports in Python?
(18 answers)
Closed 4 years ago.
I have a simple project structure like this:
➜ (venv:evernote) evernote_bear_project git:(master) ✗ tree | grep -v pyc
.
├── README.md
...
(snip)
...
├── manage.py
├── sample
│ ├── EDAMTest.py <==== here is an import that won't work
│ └── enlogo.png
└── util
├── __init__.py
├── files.py <====== This is being imported
└── test_files.py
Now I have a relative import in sample/EDAMTest.py:
from ..util.files import *
When I try to run python sample/EDAMTest.py from project root folder in command line, I get an error saying:
ValueError: attempted relative import beyond top-level package
I know this has been asked many times, but I still don't get it.
Since I'm running the script from the project root, in my understanding Python should be able to "know" that when I try to import from ..util.files import *, that it should go up one directory, no?
EDIT
Thanks for all the answers.
So what I understand from the link above is this:
I was running the module sample/EDAMTest.py directly via python sample/EDAMTest.py, and that meant
that the __name__ of the module was __main__
and now the path (sample/) was my "root" so to speak.
So now Python searches only this path and any path that's below it for modules / packages. Hence the error message attempted relative import _beyond top-level package_, so it cannot go one more level up, because it is at the root already.
Also Python cannot look one level "up", since this syntax from ..util.files import * does not go up a level in directory, but in a list of modules / packages it keeps on the "import search path" (sys.path)?
Is that correct?
sys.path.append() is a tweak, if your directory structure is fixed and there is nothing you can do about it.
Otherwise, you can try rearranging the folders. The easiest is moving util under sample, another option is making both of the folders psrt of a larger package.
Also import * is not ensorsed.
The relative import syntax is for importing other modules from the same package, not from the file system.
Depending on what you want, you could...
Fix the package so that the relative import works
Put __init__.py files in the project root and sample directory and run the script from one level up. This doesn't seem like what you want.
Tell python where to find the package
Set the PYTHONPATH environment variable so that python can find the package.
PYTHONPATH='.':$PYTHONPATH python samples/EDAMTest.py
Install util so that python can find it
Add a setup script and use it to install the util package and avoid setting PYTHONPATH.
"The relative import syntax is for importing other modules from the same package, not from the file system.", This is right as stated by George G.
Put __init__.py in your subfolders, which will make them package.
__init__.py can be an empty file but it is often used to perform
setup needed for the package(import things, load things into path, etc).
However you can import File into your __init__.py to make it
available at the package level:
# in your __init__.py
from util import files
# now import File from util package
from util import files
of if you want to import some specific method or class, you can do
from util.files import some_function
Another thing to do is at the package level make util/modules
available with the __all__ variable. When the interpeter sees
an __all__ variable defined in an __init__.py it imports the
modules listed in the __all__ variable when you do:
from package import *
__all__ is a list containing the names of modules that you want to be
imported with import * so looking at our above example again if we
wanted to import the submodules in util the all variable
in util/init.py would be:
__all__ = ['files', 'test_files']
With the __all__ variable populated like that, when you perform
from util import *
it would import files and test_files.
I am new to python and i am having some issues with packages and imports.
My structure is as follows:
src/
base/
packcore/
__init__.py
utils/
__init__.py
util_1.py
util_2.py
helpers/
__init__.py
helper_1.py
helper_2.py
some_script.py
app.py
packcore is an external package that has been installed using pip and put into the --target=base.
some of the helpers in the packcore uses some of the utils.
From app.py i want to be able to import a helper/util.
But when i use from base.packcore.utils import some_util i get an error saying that no module named packcore from inside the helper/util
and if i do from packcore.utils import some_util i get an error no module named packcorefrom theapp.py`
help would be much appreciated :)
If you add an __init__.py to base/, you can make it a Python package to import from. You also need to make the parent a package (Which is currently called src) so it is actually a sibling module, rather than many isolated modules.
From there, you can either do an absolute import from the main package:
from src.base.packcore.helpers import helper_1
Or relative (Assuming you are in some_script.py or app.py):
from .base.packcore.helpers import helper_1
I want to have this structure for my project:
requirements.txt
README.md
.gitignore
project/
__init__.py
project.py
core/
__init__.py
base.py
engines/
__init__.py
engine1.py
engine2.py
utils/
__init__.py
refine_data.py
whatever.py
The application is run from project/project.py. However, I constantly get import errors when using relative or absolute imports.
Both engines need to import from project.core.base, the utils need to import from project.core.base as well, and project.py (the main file ran) needs to be able to import from engines.
Absolute imports don't work:
# engines/engine1.py
from project.core.base import MyBaseClass
which gives the error:
ImportError: No module named project.core.base
But if I try a relative import instead
# engines/engine1.py
from ..core.base import MyBaseClass
I get:
ValueError: Attempted relative import beyond toplevel package
I've seen other projects on Github structured similarly, but this seems to cause all sorts of problems. How do I get this to work?
Take a look at your sys.path. It's likely that the top project directory is in the python path, and it sees your sub-packages (ie. utils, engines, etc.) as separate packages, which is why it's giving you an error that you're trying to import from outside your package when doing relative imports, and absolute imports don't work because it can't find the top project directory because it's not under any of the python paths.
The directory above the top project directory is what needs to be added to the python path.
Ex.
/path/is/here/project/core/...
# Add this to the PYTHONPATH
/path/is/here
Try to use these imports:
engine1.py:
from core import base
refine_data.py:
from core import base
project.py
from engines import engine1
if you use pycharm mark project directory as sources root and then try to run project.py. If you don't use pycharm you can run project.py by going to project directory and running command:
python project.py