I want import file from main folder to subsubfoler and use functions from there.
start.py
database.py
cogs/
__init__.py
Utility/
__init__.py
utility.py
Python can't import from database.py to utility.py and I don't know how can it works.
I used:
from ...database import Database
from database import Database
from ... import Database
How can I solved this problem? In future I add some folders in cogs.
The problem
Python needs to know where to look for importable modules, and it searches sys.path for them. This can be manipulated from within python, or by setting the environment variable PYTHONPATH (examples are to follow). By default python adds the entrypoint's directory to sys.path at the first position (sys.path is a list), so while running scripts from the base directory would usually work with absolute imports nested under, running from a subdirectory and trying to relatively import upwards - will not. From python's docs regarding modules:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "main", modules intended for use as the main module of a Python application must always use absolute imports.
This means that start.py will work if it import cogs (absolute), but neither start.py nor utility.py will work if they from . [or .., or ...] import cogs (relative).
How to make it work
Consider the following directory structure (unchanged from the question, just placed under /tmp/kacper/):
/tmp/kacper/
├── start.py
├── database.py
└── cogs
├── __init__.py
└── Utility
├── __init__.py
└── utility.py
__init__.py files are empty, other file contents are as follows;
/tmp/kacper/database.py:
def Database():
print("Function Database in file database.py")
/tmp/kacper/start.py:
#!/usr/bin/env python
from cogs.Utility.utility import Database
print("Done importing, now to start.py's code:")
Database()
print("Done with start.py's code")
/tmp/kacper/cogs/Utility/utility.py:
#!/usr/bin/env python
if __name__ == "__main__":
import sys
import os
if 'PYTHONPATH' not in os.environ:
print("No PYTHONPATH in env, adding project's root")
this_file_dir = os.path.dirname(os.path.abspath(__file__))
base = os.path.join(this_file_dir, "..", "..")
sys.path.append(base)
else:
print("Found PYTHONPATH in env")
else:
print("utility.py is imported, not touching sys.path")
from database import Database
if __name__ == "__main__":
print("In second main of utility.py")
Database()
print("Done second main of utility.py")
Running
Now, as start.py resides in the base of the project, it just works:
$ /tmp/kacper/start.py
utility.py is imported, not touching sys.path
Done importing, now to start.py's code:
Function Database in file database.py
Done with start.py's code
But when directly calling utility.py, we have to either specify PYTHONPATH:
$ PYTHONPATH="/tmp/kacper" /tmp/kacper/cogs/Utility/utility.py
Found PYTHONPATH in env
In second main of utility.py
Function Database in file database.py
Done second main of utility.py
or manipulate sys.path from within the script:
$ /tmp/kacper/cogs/Utility/utility.py
No PYTHONPATH in env, adding project's root
In second main of utility.py
Function Database in file database.py
Done second main of utility.py
If PYTHONPATH is not set correctly (or there's no internal handling of sys.path), this will fail; for example:
$ PYTHONPATH='/tmp' /tmp/kacper/cogs/Utility/utility.py
Found PYTHONPATH in env
Traceback (most recent call last):
File "/tmp/kacper/cogs/Utility/utility.py", line 18, in <module>
from database import Database
ModuleNotFoundError: No module named 'database'
Adding more stuff under cogs
Once python knows where to import from, this should be fairly simple. Consider the following directory structure:
/tmp/kacper/
├── database.py
├── start.py
└── cogs
├── __init__.py
├── Utility
│ ├── __init__.py
│ └── utility.py
└── newCogsFolder
├── __init__.py
├── newCogsFunctionFile.py
└── new_cogs_subfolder
├── __init__.py
└── new_cogs_subfunction_file.py
/tmp/kacper/cogs/newCogsFolder/newCogsFunctionFile.py:
def newCogsFunction():
print("new cogs function")
/tmp/kacper/cogs/newCogsFolder/new_cogs_subfolder/new_cogs_subfunction_file.py:
def new_cogs_subfunction():
print("new cogs sub function")
Then you can add to utility.py (after the first if clause):
from cogs.newCogsFolder.new_cogs_subfolder.new_cogs_subfunction_file import new_cogs_subfunction
from cogs.newCogsFolder.newCogsFunctionFile import newCogsFunction
and to start.py you can mix and match (if it makes sense):
from cogs.Utility.utility import newCogsFunction
from cogs.newCogsFolder.new_cogs_subfolder.new_cogs_subfunction_file import new_cogs_subfunction
Notes:
It usually is not a good idea to execute code during imports, and it is an even worse idea to print during them. The above code is just an example to demo the different cases.
For convenience, the files start.py and utility.py are set as executables in the above example. The result would be no different with omitting the #!/usr/bin/env python line, and running python /tmp/kacper/start.py or python /tmp/kacper/cogs/Utility/utility.py.
In the section Adding more stuff under cogs; the directory, file, and function names intentionally don't follow PEP8 - to show that naming schemes don't matter, as long as python can import them. Hint: don't use hyphens (-) in such names.
In the section Adding more stuff under cogs; While showing that it is possible for start.py to mix between direct imports and importing from utility.py, this is a pattern that is best avoided. For clarity, maintainability, and continuity - one method should be chosen, according to the specific use-case.
I've had similar problems with imports in the past and thus I've create a new import library that gives the programmer more control over their imports: https://github.com/ronny-rentner/ultraimport
It allows you to do file system based imports that will always work, no matter how you run your code, no matter what is in your sys.path and it does not care about init files.
With ultraimport, your utility.py could look like this:
import ultraimport
Database = ultraimport('__dir__/../../database.py', 'Database')
Related
I have a directory structure with 2 basic python files inside seperate directories:
├── package
│ ├── subpackage1
│ │ └── module1.py
└── subpackage2
└── module2.py
module1.py:
def module1():
print('hello world')
module2.py:
from ..subpackage1.module1 import module1
module1()
When running python3 module2.py I get the error: ImportError: attempted relative import with no known parent package
However when I run it with the imports changed to use sys.path.append() it runs successfully
import sys
sys.path.append('../subpackage1/')
from module1 import module1
module1()
Can anyone help me understand why this is and how to correct my code so that I can do this with relative imports?
To be considered a package, a Python directory has to include an __init__.py file. Since your module2.py file is not below a directory that contains an __init__.py file, it isn't considered to be part of a package. Relative imports only work inside packages.
UPDATE:
I only gave part of the answer you needed. Sorry about that. This business of running a file inside a package as a script is a bit of a can of worms. It's discussed pretty well in this SO question:
Relative imports in Python 3
The main take-away is that you're better off (and you're doing what Guido wants you to) if you don't do this at all, but rather move directly executable code outside of any module. You can usually do this by adding an extra file next to your package root dir that just imports the module you want to run.
Here's how to do that with your setup:
.
├── package
│ ├── __init__.py
│ ├── subpackage1
│ │ └── module1.py
│ └── subpackage2
│ └── module2.py
└── test.py
test.py:
import package.subpackage2.module2
You then run test.py directly. Because the directory containing the executed script is included in sys.path, this will work regardless of what the working directory is when you run the script.
You can also do basically this same thing without changing any code (you don't need test.py) by running the "script" as a module.
python3 -m package.subpackage2.module2
If you have to make what you're trying to do work, I think I'd take this approach:
import os, sys
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
from subpackage1.module1 import module1
module1()
So you compute in a relative way where the root of the enclosing package is in the filesystem, you add that to the Python path, and then you use an absolute import rather than a relative import.
There are other solutions that involve extra tools and/or installation steps. I can't think why you could possibly prefer those solutions to the last solution I show.
By default, Python just considers a directory with code in it to be a directory with code in it, not a package/subpackage. In order to make it into a package, you'll need to add an __init__.py file to each one, as well as an __init__.py file to within the main package directory.
Even adding the __init__.py files won't be enough, but you should. You should also create a setup.py file next to your package directory. Your file tree would look like this:
├── setup.py
└── package
├── __init__.py
└── subpackage1
│ ├── __init__.py
│ └── module1.py
└── subpackage2
├── __init__.py
└── module2.py
This setup.py file could start off like this:
from setuptools import setup
setup(
name='package',
packages=['package'],
)
These configurations are enough to get you started. Then, on the root of your directory (parent folder to package and setup.py), you will execute next command in you terminal pip install -e . to install your package, named package, in development mode. Then you'll be able to navigate to package/subpackage2/ and execute python module2.py having your expected result. You could even execute python package/subpackage2/module2.py and it works.
The thing is, modules and packages don't work the same way they work in another programming languages. Without the creation of setup.py if you were to create a program in your root directory, named main.py for example, then you could import modules from inside package folder tree. But if you're looking to execute package\subpackage2\module2.py.
If you want relative imports without changing your directory structure and without adding a lot of boilerplate you could use my import library: ultraimport
It gives the programmer more control over their imports and lets you do file system based relative or absolute imports.
Your module2.py could then look like this:
import ultraimport
module1 = ultraimport('__dir__/../subpackage1/module1.py')
This will always work, no matter how you run your code or if you have any init files and independent of sys.path.
This question already has answers here:
Relative imports in Python 3
(31 answers)
Closed 6 years ago.
I read a lot of answers related to the question I am asking, but still I do not understand how to make possible this thing I am trying.
So let's go to the point. I will report a simplified version of my application.
Suppose I have a main folder called project and inside it a src main package containing three subpackages:
clustering (containing a file: clustering.py)
parser (containing a file: parser.py)
support_class (containing a file: myClass.py)
In each folder, except for the project one, there is a __init__.py
Now, the python scripts contained in the clustering and parser package should use both the myclass.py contained in support_class.
I tried relative imports, but they do not works because I would like to run the scripts contained in the clustering and parser package directly and I do not want to use the -m option.
Es. python parser.py [arguments]
Example of relative import I used is:
from ..supportClass import myClass
I tried to add the package path to the sys.path but something is not working because it still tells me that it can't find the module.
Es.
sys.path.insert(0, "~/project/src")
from support_class import myClass.py
Can anyone suggest the best way to do this in python 2.7?
If I could avoid the sys.path option it would be great because I do not like this solution that much.
Thanks in advance.
Let's start from your project's folder architecture:
MyProject/
└── src
├── clustering
│ ├── __init__.py
│ └── clustering.py
├── parser
│ ├── __init__.py
│ └── parser.py
├── support_class
│ ├── __init__.py
│ └── support.py
└── main.py
If I'm not mistaken, your issue is that you want to import support.py from within parser.py and clustering.py and being able to run those two independently if needed. Two words for you:
Conditional imports
(And one more, after finding a real other solution ;): PYTHONPATH)
With the assumption that your scripts have a if __name__ == "__main__": section to run your tests, you can simply have the following as their imports:
clustering.py & parser.py:
if __name__ == "__main__":
import sys
import os
PACKAGE_PARENT = '..'
SCRIPT_DIR = os.path.dirname(os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__))))
sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))
from support_class.support import Support
else:
from support_class.support import Support
main.py:
from support_class.support import Support
Then, python clustering.py and python parser.py to your heart's content!
Which makes this a duplicate of https://stackoverflow.com/a/16985066/3425488
First, you have to create an __init __.py (two = "_", before and after, no spaces) file inside the directory where you have your actual package.
Second, you want to simply call your package from the python script where you are import to.
e.g.:
my_script.py #your script where you want to include/import your package
my_clustering_dir # directory containing files for package
my_clustering.py # file should be inside my_clustering_dir
"__init __".py # make sure this file is inside my_clustering_dir only (it can be empty)
Now, you can go to my_script.py. Then, you can add the following:
from my_clustering_dir import my_clustering #no extension (.py) needed
When you call a python script like this
python parser.py
That module is loaded as __main__, not parser.parser. It won't be loaded as part of any package, so it can't do relative imports. The correct way to do this is to create a separate script that is only used as the main script and not use any of your module scripts as main scripts. For example, create a script like this
main.py
from parser import parser
parser.main()
Then you can run python /path/to/main.py and it will work.
Also, a note on your package layout. In your example, parser and clustering and support_class aren't subpackages, they are top-level packages. Typically, if you have a package named project and you're using a src directory, your layout would look like this:
/project
setup.py
/src
/project
__init__.py
/clustering
__init__.py
/parser
..
Alternatively, if you're building an actual python package with a setup.py script, you can use the console_scripts entry point in setuptools to generate the script automatically.
setup(
...
entry_points = {
'console_scripts': ['myparser=project.parser:main'],
}
...
)
I'm building a set of Python modules that depend on each other.
My file directory currently looks like:
.
├── utilities
│ ├── __init__.py
│ ├── utility.py
│ ├── data.csv
├── src
│ ├── __init__.py
│ |── functions
│ ├── __init__.py
│ └── function.py
└── __init__.py
Further, function.py imports data from utilities/data.csv.
From the top-level directory (.), I run python3 src/functions/function.py.
And I receive the following import error:
Traceback (most recent call last):
File "src/functions/function.py", line 1, in <module>
from utilities.utility import UtilityFunctionA
ImportError: No module named 'utilities'
How do I properly import utilities from the function.py file? Or should I not be running nested files in the first place, and instead running files at the top-level of the directory?
The imports are successful when running this code from within PyCharm.
Silly question, but I've been unable to figure it out despite reading a lot of documentation (and Googling).
UPDATE:
Using python3 -m src.functions.function works to run the module from the command line with proper imports and with successfully loading the csv.
However, when I then run the module from within PyCharm, for instance using
Now I receive the error that OSError: File b'utilities/data.csv' does not exist
Is there any way to setup my module to run both from within PyCharm and also from the command line?
If you want to be able to do - from utilities.utility import UtilityFunctionA - from within function.py , when running from the top level directory - . . You need to run the python file as a module , using -m option. Example -
python3 -m src.functions.function
But The above method only works if you always run from the . directory (top-level directory) . I always either add the top-level directory manually into the PYTHONPATH environment variable.
Or use os.path and relative path to add it to sys.path programmatically using __file__. Example -
import sys
import os.path
path_to_top = os.path.abspath(os.path.join(os.path.dirname(__file__),'..','..'))
sys.path.append(path_to_top)
After this do the import. If the directory structures would always remain the same, this would work even in other systems, without having to set any environment variable.
As per the updated requirements -
Now I receive the error that OSError: File b'utilities/data.csv' does not exist
Is there any way to setup my module to run both from within PyCharm and also from the command line?
For reading files, Python uses the current working directory as the start point , and all relative paths are resolved relative to the current working directory. It is never a good idea to rely on the current working directory to be a particular directory as we can run python scripts from anywhere using absolute path to the script. For loading files as well, we can use os.path and relative paths to create the absolute path to the file. Example -
import os.path
path_to_datacsv = os.path.abspath(os.path.join(os.path.dirname(__file__),'..','..','utilities,'data.csv'))
This imports properly:
python3 -m src.functions.function
Based on How to do relative imports in Python?
There's something that's bothering me about imports in packages.
Imagine I have the following directory structure:
pack
├── __init__.py
├── sub1
│ ├── __init__.py
│ └── mod1.py
└── sub2
├── __init__.py
└── mod2.py
Inside mod1.py I have the following code to import mod2.py:
# mod1.py
import pack.sub2.mod2
pack.sub2.mod2.helloworld()
I have a main.py file in the directory containing pack that imports pack/sub1/mod1.py
How does mod1.py have access to pack? pack is not in the same directory as mod1.py. Does python automatically add the topmost package to sys.path?
You can investigate this by inspecting sys.path in an interactive interpreter. What you'll find is that the first element of it is the location of the script the interpreter was told to run. This means that when you run your script at the top level (the location of the pack package), that location is added to sys.path automatically. It doesn't have anything to do with the actual package structure, so if you ran mod1.py as a script you would have things break (this is probably why you put your script at the top level!).
Note that in Python 2, you also have the issue of implicit relative imports, which doesn't impact the issue you're asking about, but might come up if you had a few more modules involved. If you added mod3.py to sub1, you could import it from mod1 with just import mod3, with the pack.sub1 prefix being figured out implicitly. This implicit behavior is generally considered a bad thing, and in Python 3 such implicit relative imports are not allowed (you can also disable them in Python 2 with from __future__ import absolute_import). To import pack.sub1.mod3 from pack.sub1.mod1 you'd need to either name it in full, or use an explicit relative import: from . import mod3
To tie this relative import business back to your question, if you wanted to avoid relying on pack being part of sys.path (or, more realistically, protect against changes to pack's name), you could modify your import of mod2 from mod1 to be an explicit relative import. Just use from .. import sub2.mod2.
How do I import a module(python file) that resides in the parent directory?
Both directories have a __init__.py file in them but I still cannot import a file from the parent directory?
In this folder layout, Script B is attempting to import Script A:
Folder A:
__init__.py
Script A:
Folder B:
__init__.py
Script B(attempting to import Script A)
The following code in Script B doesn't work:
import ../scriptA.py # I get a compile error saying the "." is invalid
You don't import scripts in Python you import modules. Some python modules are also scripts that you can run directly (they do some useful work at a module-level).
In general it is preferable to use absolute imports rather than relative imports.
toplevel_package/
├── __init__.py
├── moduleA.py
└── subpackage
├── __init__.py
└── moduleB.py
In moduleB:
from toplevel_package import moduleA
If you'd like to run moduleB.py as a script then make sure that parent directory for toplevel_package is in your sys.path.
From the docs:
from .. import scriptA
You can do this in packages, but not in scripts you run directly. From the link above:
Note that both explicit and implicit relative imports are based on the
name of the current module. Since the name of the main module is
always "__main__", modules intended for use as the main module of a
Python application should always use absolute imports.
If you create a script that imports A.B.B, you won't receive the ValueError.
If you want to run the script directly, you can:
Add the FolderA's path to the environment variable (PYTHONPATH).
Add the path to sys.path in the your script.
Then:
import module_you_wanted