How can I import packages and modules from different places in my project without resetting the PYTHONPATH (because that seems like a 'hacky' solution).
For example if I have the following directory structure:
proj
├── __init__.py
├── important_file.py
└── subdirectory/
├── __init__.py
└── script.py
How can I import important_file.py from inside script.py? Can I somehow specify in my project that proj is my root directory, so that I can import important file with import proj.important_file?
You can use
import sys
sys.path.append('/path/to/root/directory')
to import from whatever directory you like. At least, this worked for me in the past. Maybe there are even less "hacky" solutions :)
I tend to prefer "pythonic" solutions, not necessarily what #Gijs suggested which I would call more of a work around.
So this is something I haven't done much of but when I've needed to do it, I've spent a lot of time on it. Take a look at this my repo. In it, I have a unit testing directory, which I import core.py into test_core.py.
The project folder is structured as follows.
SNA (this is called first-repo on git, but SNA on my machine)
├── bin/
└── sna/
├── core.py
└── test/
└── test_core.py
In text_core.py, I import my functions from core.py. I do this by using the following line.
from SNA.sna.core import *
So a general format is this.
from PROJECTNAME.DIR.FILE import * (or specific functions)
Hopefully this helps!
You can use relative imports
# inside subdirectory/script.py
from .. import important_file
but really the best solution is to add it to your PYTHONPATH and do
import proj.important_file
Related
I'm trying to develop custom ansible modules using the VSCode Debugger and have run into an import problem.
My ansible role structure looks like this:
.
├── defaults
├── library
│ └── dl_script
├── meta
├── module_utils
│ ├── dl_script2
│ └── dl_script3
├── tasks
├── templates
└── vars
The way ansible works is that the library folder should be where we define custom modules, with the module_utils folder the location for supporting libraries.
When I run my role with ansible it works fine because the ansible packages is supporting these libraries and makes them available to the module, in this case dt_script.
When I try to run this in vscode it falls over.
from ansible.module_utils.dt_script2 import DTScript2
from ansible.module_utils.dt_script3 import DTScript3
Error:
No module named 'ansible.module_utils.dt_script2'
What I'd like to do is figure out a way to import the code from the module_utils folder and make it do so only when it can't do it the ansible way.
I've had a go at doing this myself, but with no success:
try:
from ansible.module_utils.dt_script2 import DTScript2
from ansible.module_utils.dt_script3 import DTScript3
except:
from ..module_utils.dt_script2 import dt_script2
from ..module_utils.dt_script3 import dt_script3
Error:
attempted relative import with no known parent package
EDIT1:
I have a solution, but I don't like it.
try:
from ansible.module_utils.dt_script2 import DTScript2
from ansible.module_utils.dt_script3 import DTScript3
except:
import sys
import os
# I don't like this, but it seems to be the only way to get VScode to look for these files.
sys.path.insert(0,os.path.dirname(os.path.abspath(__file__)).replace('library', 'module_utils'))
from dt_script2 import *
from dt_script3 import *
The reason this works is the by default python only searches the dir and subdir of the place you are running the script from, or certain package locations, like pyenv if you have it. What I'm doing here is amending the path list object to include another path to check from.
Reason this is bad in my mind is that this relies on path manipulation, which makes this very reliant on the host pathing not being really odd. I'd much prefer a safer solution for this, but this is only for when people want to use vscode to debug it.
If anyone has a better solution I'd love to see it.
This is the structure of my project
final
├── common
├── __init__.py
├── batch_data_processing.py
├── processing_utility.py
├── post_processing.py
├── Processing_raw_data
├── batch_process_raw_data.py
so i want to import from common.batch_data_processing in batch_process_raw_data.py
but when I try it I get ModuleNotFoundError: No module named 'common'
is there a way to import this module without the need to install it ?
Note : this is intended to be used by "non python users"
here is pictures to better discribe the problem.
Add the following code above your import code to indicate the path:
# The following is a relative path,
# it can also be replaced with the absolute path
# of the directory where common is located.
# sys.path.append("C:\\Users\\Admin\\Desktop\\Final")
import sys
sys.path.append("./")
When all your scripts are in the same folder, importing modules is almost impossible to go wrong. If you need to import scripts from external folders, you can specify the path using the above method.
I've read many docs over the last few days about relative Python imports but run into a struggle with the following folder structure:
parent_folder
├── subfolder1
│ └── __init__.py
│ └── file_1.py
├── subfolder2
│ └── __init__.py
│ └── file_2.py
│
└ __init__.py (parent folder has an init in it)
In file_2.py I would like to access the functions in file_1.py. I've tried adding the following to file_2.py but none seem to work:
1. from ..subfolder1 import file_1 #ImportError: attempted relative import with no known parent package
2. import parent_folder.subfolder1.file_1 #ModuleNotFoundError: No module named 'parent_folder'
3. from parent_folder.subfolder1 import file_1 #ModuleNotFoundError: No module named 'parent_folder'
I'm really lost right now and can't seem to understand why this is occuring. I've probably read 10 different guides on relative imports now and still can't figure out why.
Note, if I put file_2.py inside parent_folder, and then add import subfolder1.file1 it imports well, but I can't move file_2.py from it's position or use sys.path.append()
Does anyone with more module experience than I have any insight? Thank you!
The answers advising messing with the sys path are wrong - unfortunately this advice is floating over the web causing infinite frustration and crashes (good) to subtle bugs (bad).
The correct answer is running your script using the -m switch from the parent folder of the top most package. So if this parent_folder is a package as it looks it is and you want to run file_1.py you should
$ python -m parent_folder.subfolder1.file_1
any of the three imports would work then
Change Path
Make sure you change sys.path before you start importing anything so you don't get an error when importing
So, begin with this:
import os, sys
path = os.path.join(os.path.dirname(__file__), os.pardir)
sys.path.append(path)
In my case I did this
import sys
sys.path.insert(0, '..')
then
from parent_folder.subfolder1.file_1 import --the-function-needed--
I realize there are a slew of posts on SO related to Python and imports, but it seems like a fair number of these posts are asking about import rules/procedures with respect to creating an actual Python package (vs just a project with multiple directories and python files). I am very new to Python and just need some more basic clarification on what is and is not possible with regard to access/importing within the context of multiple py files in a project directory.
Let's say you have the following project directory (to be clear, this is not a package that is somewhere on sys.path, but say, on your Desktop):
myProject/
├── __init__.py
├── scriptA.py
└── subfolder
├── __init__.py
└── scriptB.py
└── subsubfolder
├── __init__.py
└── scriptC.py
└── foo.py
Am I correct in understanding that the only way scriptC.py could import and use methods or classes within scriptB.py if scriptC.py is run directly via $ python scriptC.py and from within the subsubfolder directory is if I add the parent directory and path to scriptB.py to the Python path at runtime via sys.path ?
It is possible, however, for scriptC.py to import foo.py or for scriptB.py to import scriptC.py or foo.py without dealing with sys.path, correct? Adjacent py files and py files in subdirectories are accessible just by using relative import paths, you just can't import python scripts that live in parent or sibling directories (without using sys.path) ?
What's Possible
Anything.
No, really. See the imp module, the the imputil module -- take a look at how the zipimport module is written if you want some inspiration.
If you can get a string with your module's code in a variable, you can get a module into sys.modules using the above, and perhaps hack around with its contents using the ast module on the way.
A custom import hook that looks in parent directories? Well within the range of possibilities.
What's Best Practice
What you're proposing isn't actually good practice. The best-practice approach looks more like the following:
myProject/
├── setup.py
└── src/
├── moduleA.py
└── submodule/
├── __init__.py
├── moduleB.py
└── subsubmodule/
├── __init__.py
└── moduleC.py
Here, the top of your project is always in myProject/src. If you use setup.py to configure moduleA:main, submodule.moduleB:main and submodule.subsubmodule.moduleC:main as entry points (perhaps named scriptA, scriptB and scriptC), then the functions named main in each of those modules would be invoked when the user ran the (automatically generated by setuptools) scripts so named.
With this layout (and appropriate setuptools use), your moduleC.py can absolutely import moduleA, or import submodule.moduleB.
Another approach, which doesn't involve entrypoints, to invoke the code in your moduleC.py (while keeping the module's intended hierarchy intact, and assuming you're in a virtualenv where python setup.py develop has been run) like so:
python -m submodule.subsubmodule.moduleC
I am having an import error problem in a test script. It looks like its something to do with the directory structure.
I have the following folder structure:
A
├── F1
│ ├── __init__.py
│ └── Src
│ └── F2
│ └── __init__.py
└── tests1
└── tests1
└── test_script.py
A/F1/Src/F2
F1 has "__init__py" in its level
F2 has "__init__.py" in its level
In the same level as F1, there is another folder "tests1"
tests1/tests1/test_script.py
in test_script.py, I have a line which says
from F1.src.F2 import C
With the above, I get an error saying, no module named "F1.src.F2"
Does someone know what is going on here?
from F1.src.F2 import C is an absolute import. In order for it to work, "F1" has to be located somewhere on your Python path (sys.path). Generally this also includes the current directory if you ran Python on the command line.
So if the A directory is not one of the directories on your Python path and is not your current working directory, there is no reason the import would work.
Module names are case sensitive. You have Src in one place and src in another, but I'm not sure that reflects your actual directory structure or just what you typed here.
Using a relative import will not work if you are running test_script.py as a script (Which is what it sounds like.) So, what you really want to do is make sure that either you run the script from the A directory, or go whole hog, make your project into a legit package with a setup.py and use a test runner such as tox.
I just had to create a shared library with the "egg" file.
As simple as that but it occurred to me late!