I have the following structure in my Python project:
project
├── subdir
│ ├── __init__.py
│ └── script_to_run.py
├── __init__.py
└── functions.py
In script_to_run.py file, I want to import a function from top-level functions.py file as
from functions import function_to_import
When I try to run the script from either root directory (project) or its subdirectory (subdir), I get the following error:
ModuleNotFoundError: No module named 'functions'
How do I import it then?
I've seen several approaches, including messing with your sys.path or installing your local package through pip, but that seems like too much work for something this simple, right?
Python packages are a great thing, but come at a cost: you cannot directly run a file lying deep in the package.
If the file is imported from the package, the correct way would be:
from ..functions import function_to_import
If the file is intended to be launched as a top level script, it should not be in the package. If you really want to go that way, the solution is to add the project directory to sys.path and use:
from functions import function_to_import
but it is an anti-pattern... You have been warned...
Related
I have a directory structure with 2 basic python files inside seperate directories:
├── package
│ ├── subpackage1
│ │ └── module1.py
└── subpackage2
└── module2.py
module1.py:
def module1():
print('hello world')
module2.py:
from ..subpackage1.module1 import module1
module1()
When running python3 module2.py I get the error: ImportError: attempted relative import with no known parent package
However when I run it with the imports changed to use sys.path.append() it runs successfully
import sys
sys.path.append('../subpackage1/')
from module1 import module1
module1()
Can anyone help me understand why this is and how to correct my code so that I can do this with relative imports?
To be considered a package, a Python directory has to include an __init__.py file. Since your module2.py file is not below a directory that contains an __init__.py file, it isn't considered to be part of a package. Relative imports only work inside packages.
UPDATE:
I only gave part of the answer you needed. Sorry about that. This business of running a file inside a package as a script is a bit of a can of worms. It's discussed pretty well in this SO question:
Relative imports in Python 3
The main take-away is that you're better off (and you're doing what Guido wants you to) if you don't do this at all, but rather move directly executable code outside of any module. You can usually do this by adding an extra file next to your package root dir that just imports the module you want to run.
Here's how to do that with your setup:
.
├── package
│ ├── __init__.py
│ ├── subpackage1
│ │ └── module1.py
│ └── subpackage2
│ └── module2.py
└── test.py
test.py:
import package.subpackage2.module2
You then run test.py directly. Because the directory containing the executed script is included in sys.path, this will work regardless of what the working directory is when you run the script.
You can also do basically this same thing without changing any code (you don't need test.py) by running the "script" as a module.
python3 -m package.subpackage2.module2
If you have to make what you're trying to do work, I think I'd take this approach:
import os, sys
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
from subpackage1.module1 import module1
module1()
So you compute in a relative way where the root of the enclosing package is in the filesystem, you add that to the Python path, and then you use an absolute import rather than a relative import.
There are other solutions that involve extra tools and/or installation steps. I can't think why you could possibly prefer those solutions to the last solution I show.
By default, Python just considers a directory with code in it to be a directory with code in it, not a package/subpackage. In order to make it into a package, you'll need to add an __init__.py file to each one, as well as an __init__.py file to within the main package directory.
Even adding the __init__.py files won't be enough, but you should. You should also create a setup.py file next to your package directory. Your file tree would look like this:
├── setup.py
└── package
├── __init__.py
└── subpackage1
│ ├── __init__.py
│ └── module1.py
└── subpackage2
├── __init__.py
└── module2.py
This setup.py file could start off like this:
from setuptools import setup
setup(
name='package',
packages=['package'],
)
These configurations are enough to get you started. Then, on the root of your directory (parent folder to package and setup.py), you will execute next command in you terminal pip install -e . to install your package, named package, in development mode. Then you'll be able to navigate to package/subpackage2/ and execute python module2.py having your expected result. You could even execute python package/subpackage2/module2.py and it works.
The thing is, modules and packages don't work the same way they work in another programming languages. Without the creation of setup.py if you were to create a program in your root directory, named main.py for example, then you could import modules from inside package folder tree. But if you're looking to execute package\subpackage2\module2.py.
If you want relative imports without changing your directory structure and without adding a lot of boilerplate you could use my import library: ultraimport
It gives the programmer more control over their imports and lets you do file system based relative or absolute imports.
Your module2.py could then look like this:
import ultraimport
module1 = ultraimport('__dir__/../subpackage1/module1.py')
This will always work, no matter how you run your code or if you have any init files and independent of sys.path.
Although I think this should be pretty simple I still can't make it run.
I have following folder structure:
├── apartment
│ ├── src
│ ├── train_model
│ ├── __init__.py
│ ├── train_model.py
│ ├── utils.py
│ ├── interference.py
│ └── __init__.py
In utils.py I tried:
from src.interference import create_sample
Error: ModuleNotFoundError: No module named 'src'
from .interference import create_sample
Error: ImportError: attempted relative import with no known parent package
from interference import create_features_sample
ModuleNotFoundError: No module named 'interference'
What is the way to make it work? I am not a huge fan of non-pythonic ways as it looks dirty.
The structure starting with a src/ is aimed explicitly at not enabling import by from src.intereference import ..., and you should not put an __init__.py file in src/ folder.
Instead, following nice explanation and examples here: https://blog.ionelmc.ro/2014/05/25/python-packaging/, here is what I recommend:
install the package:
add a setup.py file at root of your folder (this is clearly not as hard as it seems)
maybe create a virtual environment
using pip install -e . (with trailing dot!) command
then simply import your package by from interference import ...
To answer to your primary request, you could update src/__init__.py with from intereference import create_sample, to expose this function at a higher level, then the chained import would work. However, I do not recommend this, as it renders everything very rigid.
You need to add the directory that contains interference to PYTHONPATH.
You can use OS depending path in "module search path" which is listed in sys.path. So you can easily add parent directory like following:
import sys
sys.path.insert(0, '..')
from interference import create_features_sample
Note that the previous code uses a relative path, so you must launch the file inside the same location or it will likely not work.
To launch from anywhere, you can use Path from the pathlib module.
from pathlib import Path
import sys
path = str(Path(Path(__file__).parent.absolute()).parent.absolute())
sys.path.insert(0, path)
from interference import create_features_sample
If you want add a custom path to your PYTHONPATH permanently, go to the 'site-packages' folder of your current Python environment and add the file 'custompaths.pth' in which each line should consist of a directory which will then be checked if you try to import the module.
Assuming 'src' being the module you want to import you should add the following line to the .pth file:
your_preceding_path/apartment
Error because of python can't able to find the specific file or package. Python usually find only for the child packages or files, otherwise you need to specify the absolute path.
import sys, os
sys.path.append(os.path.abspath(os.path.join('../..', 'src')))
Please refer the "How to access a module from outside your file folder in Python?"
Have you tried from ..interference import create_sample ?
Or for the whole module from .. import interference.
I've checked here, that I'm using another command, something that is missing from the question at the time of writing this. The command that I'm using is python -m src.train_model.utils from apartment folder.
Thanks to RMPR for helping me.
This problem is similar to this one.
I am trying to import * from a file classes.py. My directory is as follows
mypkg
├── main.py
├── classes.py
When I try
from classes import *
It does not recognise classes. Looking it up I saw that I should use the explicit import
from .classes import *
which does recognise classes but gives the error below when I try to run it.
ModuleNotFoundError: No module named '__main__.classes'; '__main__' is not a package
Any advice on what to do and why this is happening would be hugely appreciated.
For python to recognize a folder as a package, you need an __init__.py file in it:
mypkg
├── __init__.py
├── main.py
├── classes.py
The directory where python is invoked is also important (running from inside a package folder is different to running from outside), and there's the PYTHONPATH environment variable as well.
The python documentation as a section on import and the package system, with good explanations.
I realize there are a slew of posts on SO related to Python and imports, but it seems like a fair number of these posts are asking about import rules/procedures with respect to creating an actual Python package (vs just a project with multiple directories and python files). I am very new to Python and just need some more basic clarification on what is and is not possible with regard to access/importing within the context of multiple py files in a project directory.
Let's say you have the following project directory (to be clear, this is not a package that is somewhere on sys.path, but say, on your Desktop):
myProject/
├── __init__.py
├── scriptA.py
└── subfolder
├── __init__.py
└── scriptB.py
└── subsubfolder
├── __init__.py
└── scriptC.py
└── foo.py
Am I correct in understanding that the only way scriptC.py could import and use methods or classes within scriptB.py if scriptC.py is run directly via $ python scriptC.py and from within the subsubfolder directory is if I add the parent directory and path to scriptB.py to the Python path at runtime via sys.path ?
It is possible, however, for scriptC.py to import foo.py or for scriptB.py to import scriptC.py or foo.py without dealing with sys.path, correct? Adjacent py files and py files in subdirectories are accessible just by using relative import paths, you just can't import python scripts that live in parent or sibling directories (without using sys.path) ?
What's Possible
Anything.
No, really. See the imp module, the the imputil module -- take a look at how the zipimport module is written if you want some inspiration.
If you can get a string with your module's code in a variable, you can get a module into sys.modules using the above, and perhaps hack around with its contents using the ast module on the way.
A custom import hook that looks in parent directories? Well within the range of possibilities.
What's Best Practice
What you're proposing isn't actually good practice. The best-practice approach looks more like the following:
myProject/
├── setup.py
└── src/
├── moduleA.py
└── submodule/
├── __init__.py
├── moduleB.py
└── subsubmodule/
├── __init__.py
└── moduleC.py
Here, the top of your project is always in myProject/src. If you use setup.py to configure moduleA:main, submodule.moduleB:main and submodule.subsubmodule.moduleC:main as entry points (perhaps named scriptA, scriptB and scriptC), then the functions named main in each of those modules would be invoked when the user ran the (automatically generated by setuptools) scripts so named.
With this layout (and appropriate setuptools use), your moduleC.py can absolutely import moduleA, or import submodule.moduleB.
Another approach, which doesn't involve entrypoints, to invoke the code in your moduleC.py (while keeping the module's intended hierarchy intact, and assuming you're in a virtualenv where python setup.py develop has been run) like so:
python -m submodule.subsubmodule.moduleC
I'm building a set of Python modules that depend on each other.
My file directory currently looks like:
.
├── utilities
│ ├── __init__.py
│ ├── utility.py
│ ├── data.csv
├── src
│ ├── __init__.py
│ |── functions
│ ├── __init__.py
│ └── function.py
└── __init__.py
Further, function.py imports data from utilities/data.csv.
From the top-level directory (.), I run python3 src/functions/function.py.
And I receive the following import error:
Traceback (most recent call last):
File "src/functions/function.py", line 1, in <module>
from utilities.utility import UtilityFunctionA
ImportError: No module named 'utilities'
How do I properly import utilities from the function.py file? Or should I not be running nested files in the first place, and instead running files at the top-level of the directory?
The imports are successful when running this code from within PyCharm.
Silly question, but I've been unable to figure it out despite reading a lot of documentation (and Googling).
UPDATE:
Using python3 -m src.functions.function works to run the module from the command line with proper imports and with successfully loading the csv.
However, when I then run the module from within PyCharm, for instance using
Now I receive the error that OSError: File b'utilities/data.csv' does not exist
Is there any way to setup my module to run both from within PyCharm and also from the command line?
If you want to be able to do - from utilities.utility import UtilityFunctionA - from within function.py , when running from the top level directory - . . You need to run the python file as a module , using -m option. Example -
python3 -m src.functions.function
But The above method only works if you always run from the . directory (top-level directory) . I always either add the top-level directory manually into the PYTHONPATH environment variable.
Or use os.path and relative path to add it to sys.path programmatically using __file__. Example -
import sys
import os.path
path_to_top = os.path.abspath(os.path.join(os.path.dirname(__file__),'..','..'))
sys.path.append(path_to_top)
After this do the import. If the directory structures would always remain the same, this would work even in other systems, without having to set any environment variable.
As per the updated requirements -
Now I receive the error that OSError: File b'utilities/data.csv' does not exist
Is there any way to setup my module to run both from within PyCharm and also from the command line?
For reading files, Python uses the current working directory as the start point , and all relative paths are resolved relative to the current working directory. It is never a good idea to rely on the current working directory to be a particular directory as we can run python scripts from anywhere using absolute path to the script. For loading files as well, we can use os.path and relative paths to create the absolute path to the file. Example -
import os.path
path_to_datacsv = os.path.abspath(os.path.join(os.path.dirname(__file__),'..','..','utilities,'data.csv'))
This imports properly:
python3 -m src.functions.function
Based on How to do relative imports in Python?