Importing Module Between Directories - python

UPDATE (TO SHOW FINAL CODE)
Since this seemed hard to explain, I shared the project. For those that come to this question, you can see the full project here:
https://github.com/jeffnyman/pacumen
Calling out the files that were problematic for me:
graphical_pacman
https://github.com/jeffnyman/pacumen/blob/master/displays/graphical_pacman.py
layout
https://github.com/jeffnyman/pacumen/blob/master/mechanics/layout.py
With the __init__.py and setup.py files in place, I'm now able to run commands like these:
python displays/graphical_pacman.py
python mechanics/layout.py
All imports now resolve correctly when those commands are executed. You can see all the import statements I use in each file and the placement of the various __init__.py files.
ORIGINAL QUESTION
I cannot get what seems to be a simple thing to work: importing modules between directories. This is all in Python 3 so I don't want to have __init__.py files all over the place if I can help it, which many of the answers here suggest is the "right" way.
I have a structure like this:
project
displays
graphical_pacman.py
mechanics
layout.py
The layout.py file has a top level function called get_layout() that I want to call from graphical_pacman.py.
Going to the minimum code necessary, in graphical_pacman.py I have:
import layout
if __name__ == '__main__':
board = layout.get_layout("test_maze.lay")
Shows up fine in IDE, even autocompletes it for me. Running graphical_pacman.py gets me this:
File "displays/graphical_pacman.py", line 3, in <module>
import layout
ModuleNotFoundError: No module named 'layout'
Then I tried this:
from mechanics.layout import get_layout
if __name__ == '__main__':
board = mechanics.layout.get_layout("test_maze.lay")
Can't do that either:
File "displays/graphical_pacman.py", line 3, in <module>
from mechanics.layout import get_layout
ModuleNotFoundError: No module named 'mechanics'
I tried this:
from mechanics import layout
if __name__ == '__main__':
board = layout.get_layout("test_maze.lay")
I tried this:
from layout import get_layout
if __name__ == '__main__':
board = get_layout("test_maze.lay")
Doesn't work. Got this:
File "displays/graphical_pacman.py", line 3, in <module>
from layout import get_layout
ModuleNotFoundError: No module named 'layout'
I tried using relative imports (with the . in front of things) but that also doesn't work. I've also just tried using the * for my import (essentially importing everything). Also doesn't work. When I say "doesn't work" I get some variation of the above errors.
I've tried all of this running the command python graphical_pacman.py from within the displays directory and at the root project directory. The same errors occur each time.
I also tried using a sys.path, such as this sys.path.insert(0, '../mechanics'). I also tried variations on sys.path.append('../') based on other answers I've seen here. Again, all I get are variations on the above errors.
What am I missing?

2nd Update:
I have sent a pull request, check it out.
Here's what I did: I created a package out of the project root directory and installed it using pip. Details follow:
Files Added:
pacumen
__init__.py
setup.py
displays
__init__.py
library
__init__.py
mechanics
__init__.py
Contents of pacumen/__init__.py:
from . import displays
from . import library
from . import mechanics
Contents of pacumen/setup.py:
import setuptools
setuptools.setup(
name='pacumen',
version='0.01dev',
packages=['displays',
'library',
'mechanics',
],
author='jefferyman',
author_email='something#something.com',
description='Pacman. Duh.',
long_description=open('README.md').read(),
)
Contents of pacumen/displays/__init__.py:
from . import graphical_helpers
from . import graphical_pacman
from . import graphical_support
Contents of pacumen/library/__init__.py:
from . import structures
from . import utilities
Contents of pacumen/mechanics/__init__.py:
from . import grid
from . import layout
Changes in files:
pacumen/mechanics/layout.py:
from mechanics.grid import Grid
Make sure your virtual environment is active. (Instructions for that further down).
Finally, navigate to the project root directory and install your project root as a package:
pip install --editable . # or pip install -e . (Note the period at the end)
Now as long as you activate the virtual environment, you should not have any import problems. Do make sure you use the import statements of the style:
from mechanics.grid import Grid
Virtual Environment creation and activation:
For those reading, now is a good time to make a virtual environment if one isn't already made. If made make sure to activate it. If not, navigate to the root of the project dir (pacumen) and run
$ python -m venv venvdir
Then, once it is created, run:
<project-root>$ .venvdir\Scripts\activate # for windows.
OR
<project-root>$ source venvdir/bin/activate # for linux bash
Update: First things to check are that there are no circular imports, which means you are importing something in graphical_pacman from mechanics and something else in mechanics from graphical_pacman. Also make sure your module names don't conflict with built-in python module's names. If it's all good on these fronts,
Have you tried this?
from project.mechanics import layout
If that didn't work place __init__.py files in project, displays, mechanics.
In project/displays/__init__.py add from . import graphical_pacman.
In project/mechanics/__init__.py add from. import layout.
If neither of these worked, create a package out of this, navigate to the package root and install it in you environment with pip install -e .

I have something that works so I have an answer but the full answer is that what I'm doing can't be done in Python, as far as I can tell. Part of the problem here is that as I learned more, I realized my question was not worded well enough in terms of how I wanted things to execute.
Here's how things can work. If in graphical_pacman.py, I do this:
if __name__ == '__main__':
sys.path.append('../')
from mechanics import layout
board = layout.get_layout("test_maze.lay")
This finally works!
BUT ...
I have to run python graphical_pacman.py from within the displays directory rather than at the project root. (Meaning, I can't do this command from the project root: python displays\graphical_pacman.py.)
BUT ...
Execution still runs into an error:
Traceback (most recent call last):
File "graphical_pacman.py", line 18, in <module>
from mechanics import layout
File "../mechanics/layout.py", line 4, in <module>
from grid import Grid
ModuleNotFoundError: No module named 'grid'
And that's coming from this line in layout.py:
from grid import Grid
If I change that line to a relative import:
from .grid import Grid
NOW my python graphical_pacman.py command works when run from the displays directory.
BUT ...
Now I can't just run python layout.py, which also needs to be possible. That's apparently because the relative import is now in there, and so I get this error:
Traceback (most recent call last):
File "layout.py", line 4, in <module>
from .grid import Grid
ModuleNotFoundError: No module named '__main__.grid'; '__main__' is not a package
The problem I'm finding is I need to sometimes run files individually. You'll see mechanics\layout.py has an if __name__ == '__main__': section. So does displays\graphical_support.py. Those are necessary so people can see how things work by running those files directly. But eventually those will be used/imported by a file that's in the root of the directory that is orchestrating everything. And sometimes files (like layout) will depend on others (like grid).
So it seems the answer is that Python can't do what I'm trying to do, which is basically just seek flexibility.
I suppose I could handle all of this by just having all of the files in one directory, which seems to be the best answer here, and seems to be how other projects have done it. That seems like a bad approach but I'm weighing that against fighting with an import mechanism particularly since I've now discovered that the mechanism works in different ways depending on execution.

Related

File import issue - python Django

I have a file in my management/commands folder and I am trying to import my models into the report.py folder. However, I get the following error.
Traceback (most recent call last):
File "c:\Users\Timmeh\source\Python\Django Projects\env\topxgym\members\management\commands\report.py", line 2, in <module>
from members.models import ActiveMember
ModuleNotFoundError: No module named 'members'
I have a active.py file in that same folder with the same import and it works. However, the active.py is run from python manage.py active. The report.py I am just compiling/running it (not running it from python manage.py). That is the only difference.
the file structure.
Please don't forget to add __init__.py for every folder.
More here:
https://docs.python.org/3/reference/import.html#regular-packages
Try to made relative import.
from ..models import ActiveMember
if it works, this answer can help:
Absolute imports in python not working, relative imports work
Try to check your project settings.
Have you add yors members app in settings.INSTALLED_APPS list?
Here is a dirty hack which should solve the issue.
and I'll also point out an alternate one if that suits your use case.
First Option
import sys
sys.path.append(r'c:\Users\Timmeh\source\Python\Django Projects\env\topxgym')
# now your import should work
from members.models import ActiveMember
This is dirty because it manipulates sys path which is not a good practice. I'm assuming this script is just a standalone script you wish to run, hence it should not be an issue overall.
Second Option
IFF this is a possibility, you could just move your report.py and keep it inside topxgym folder (at the same level as requirements.txt)
In that case too, the same import statement would work.
side note
If you're looking to access the ORM from a standalone script, it looks like you might be missing to add the django configuration in the script as recommended by the documentation.
After your import issues are fixed by either of the above two methods, if it works, great. If it doesn't it would mean that you have to add the django.setup() config and specify a settings file.

Import Modules from same directory

I'm trying to import python modules within the same directory using PyCharm which should be a very easy task. I've been struggling with this bug for a while and since I had to finish projects I regularly used the 'SourceFileLoader' from 'importlib.machinery' given the related base directory.
Since this method is quite inefficient for sharing my codes I'd like to finally deal with the problem for good.
Here one example:
Directory
C:\Users\Hallo.DESKTOP-4OVSO14\PycharmProjects\Uni\Tello_Video
contains files:
__init__.py
main.py
tello.py
tello_control_ui.py
main.py contains following code:
import tello from tello_control_ui
import TelloUI
def main():
drone = tello.Tello('', 8889)
vplayer = TelloUI(drone,"./img/")
# start the Tkinter mainloop
vplayer.root.mainloop()
if __name__ == "__main__":
main()
which produces error message:
No module named 'tello'
If i change the import lines of this script to find the overall directory 'Tello_Video' it does the trick.
from Tello_Video import tello, tello_control_ui
Still this solution is very unconvenient since a lot of open source scripts are using the direct way for scripts in the same directory.
I'm running Python 3.7 on Windows 10 using Pycharm. I've read other question concerning this matter and already tried following methods:
1.) update package setuptools
2.) creating overall directory (in this example /Tello_Video) as 'Python Package'
3.) execute Pycharm in the same directory (/Tello_Video) where python files are stored
(4.) __init__.py is empty)
Thank you for any advice!

How do I find details of this python package (__main__)? [duplicate]

Currently trying to work in Python3 and use absolute imports to import one module into another but I get the error ModuleNotFoundError: No module named '__main__.moduleB'; '__main__' is not a package. Consider this project structure:
proj
__init__.py3 (empty)
moduleA.py3
moduleB.py3
moduleA.py3
from .moduleB import ModuleB
ModuleB.hello()
moduleB.py3
class ModuleB:
def hello():
print("hello world")
Then running python3 moduleA.py3 gives the error. What needs to be changed here?
.moduleB is a relative import. Relative only works when the parent module is imported or loaded first. That means you need to have proj imported somewhere in your current runtime environment. When you are are using command python3 moduleA.py3, it is getting no chance to import parent module. You can:
from proj.moduleB import moduleB OR
You can create another script, let's say run.py, to invoke from proj import moduleA
Good luck with your journey to the awesome land of Python.
Foreword
I'm developing a project which in fact is a Python package that can be installed through pip, but it also exposes a command line interface. I don't have problems running my project after installing it with pip install ., but hey, who does this every time after changing something in one of the project files? I needed to run the whole thing through simple python mypackage/main.py.
/my-project
- README.md
- setup.py
/mypackage
- __init__.py
- main.py
- common.py
The different faces of the same problem
I tried importing a few functions in main.py from my common.py module. I tried different configurations that gave different errors, and I want to share with you with my observations and leave a quick note for future me as well.
Relative import
The first what I tried was a relative import:
from .common import my_func
I ran my application with simple: python mypackage/main.py. Unfortunately this gave the following error:
ModuleNotFoundError: No module named '__main__.common'; '__main__' is not a package
The cause of this problem is that the main.py was executed directly by python command, thus becoming the main module named __main__. If we connect this information with the relative import we used, we get what we have in the error message: __main__.common. This is explained in the Python documentation:
Note that relative imports are based on the name of the current module. Since the name of the main module is always __main__, modules intended for use as the main module of a Python application must always use absolute imports.
When I installed my package with pip install . and then ran it, it worked perfectly fine. I was also able to import mypackage.main module in a Python console. So it looks like there's a problem only with running it directly.
Absolute import
Let's follow the advise from the documentation and change the import statement to something different:
from common import my_func
If we now try to run this as before: python mypackage/main.py, then it works as expected! But, there's a caveat when you, like me, develop something that need to work as a standalone command line tool after installing it with pip. I installed my package with pip install . and then tried to run it...
ModuleNotFoundError: No module named 'common'
What's worse, when I opened a Python console, and tried to import the main module manually (import mypackage.main), then I got the same error as above. The reason for that is simple: common is no longer a relative import, so Python tries to find it in installed packages. We don't have such package, that's why it fails.
The solution with an absolute import works well only when you create a typical Python app that is executed with a python command.
Import with a package name
There is also a third possibility to import the common module:
from mypackage.common import my_func
This is not very different from the relative import approach, as long as we do it from the context of mypackage. And again, trying to run this with python mypackage/main.py ends similar:
ModuleNotFoundError: No module named 'mypackage'
How irritating that could be, the interpreter is right, you don't have such package installed.
The solution
For simple Python apps
Just use absolute imports (without the dot), and everything will be fine.
For installable Python apps in development
Use relative imports, or imports with a package name on the beginning, because you need them like this when your app is installed. When it comes to running such module in development, Python can be executed with the -m option:
-m mod : run library module as a script (terminates option list)
So instead of python mypackage/main.py, do it like this: python -m mypackage.main.
In addition to md-sabuj-sarker's answer, there is a really good example in the Python modules documentation.
This is what the docs say about intra-package-references:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.
If you run python3 moduleA.py3, moduleA is used as the main module, so using the absolute import looks like the right thing to do.
However, beware that this absolute import (from package.module import something) fails if, for some reason, the package contains a module file with the same name as the package (at least, on my Python 3.7). So, for example, it would fail if you have (using the OP's example):
proj/
__init__.py (empty)
proj.py (same name as package)
moduleA.py
moduleB.py
in which case you would get:
ModuleNotFoundError: No module named 'proj.moduleB'; 'proj' is not a package
Alternatively, you could remove the . in from .moduleB import, as suggested here and here, which seems to work, although my PyCharm (2018.2.4) marks this as an "Unresolved reference" and fails to autocomplete.
Maybe you can do this before importing the moduleļ¼š
moduleA.py3
import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
from moduleB import ModuleB
ModuleB.hello()
Add the current directory to the environment directory
Just rename the file from where you run the app to main.py:
from app import app
if __name__ == '__main__':
app.run()
import os
import sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
will solve the issue of import path issue.

Python gives a Import Error and i don't understand why

I'm still learning to code better in Python. Therefore I am trying to use some constructions in my programming. Things like name_conventions and structured packaging & modules.
That said I come across an issue which I cannot resolve, it seems.
The structure I chose is as follows:
Core
__init__.py
controller.py
Dashboard
Log
Plugins
Stats
__init__.py
controller.py
Plugin2
Plugin3
Etc..
main.py
In the Core controller.py i am trying to import the Stats(object) class.
I have tried several different ways. In my IDE (Pycharm) it works like a charm ;) When i try to start it via Command Line i get ImportErrors.
I have tried different approaches.
from Plugins.Stats.controller import Stats
from Plugins.Stats import controller
from Plugins.Stats import controller.Stats
from Plugins import Stats.controller
I also tried to put the Stats class into the Plugins.Stats.__init__.py and call it:
from Plugins.Stats import Stats
and I get this:
Traceback (most recent call last):
File "controller.py", line 4, in <module>
from Plugins.Stats.controller import Stats
ImportError: No module named 'Plugins'
The funny thing is... When i try to do the same from main.py.
It does work. So I am really confused. Am I doing something wrong?
Also are there special design patterns to handle imports?
You should add __init__.py file also in the Plugins directory so it will look like
Plugins
__init__.py
Stats
__init__.py
controller.py
You can read more about Python modules here - check out 6.4. Packages
you also can make your module you want to import visible for everyone by adding it to the pythonpath
sys.path.append("/path/to/main_folder")
before this modules are on another namespaces
You will find more info here
First put From plugin import stats, then from stats import controller and import _init_
Incase of running from the terminal it will only try to import from the current folder. You have to manually add the path of the main folder to this file using the following code at the beging of the file
import sys
sys.path.append("../")
If you want to back 2 folders add ../../ instead of ../
You should not start a submodule of a package directly as a script. This will mess up your hierarchy most of the time. Instead call it as a module:
python -m Core.controller
You can verify the following by adding this to the top of any of your scripts:
import sys
for p in sys.path:
print(repr(p))
The reason the imports work when invoked as:
python main.py
is python will add the directory of the script to the pythonpath (In this case the empty string '')
However, when running
python Core/controller.py
it will add Core to sys.path, rendering the rest of your project unimportable
You can instead use
python -m Core.controller
which'll invoke using runpy and again put the empty string on the python path.
You can read more about invocation options of the python interpreter here: https://docs.python.org/2/using/cmdline.html You can read about runpy here: https://docs.python.org/2/library/runpy.html

Understanding A Chain of Imports in Python

I know there are several similar questions, but I'm struggling to understand the error I'm getting and browsing the docs and similar questions hasn't helped yet. If anything, the similar questions make me feel like what I'm doing is right.
I have the following files:
src/main.py
from pack import pack
if __name__ == '__main__':
pack.exec("Hello Universe!")
src/pack/pack.py
import util
def exec(text):
util.write(text)
if __name__ == '__main__':
exec("Hello World!")
src/pack/util.py
def write(text):
print(text)
*src/pack/_init_.py*
EMPTY FILE
When I run python pack.py from the src/pack directory, it works (prints "Hello World!"). However when I run python main.py from the src directory I get the following exception:
Traceback (most recent call last):
File ".../src/main.py", line 1, in <module>
from pack import pack
File ".../src/pack/pack.py", line 1, in <module>
import util
ImportError: No module named util
If I change the import line in pack.py to from . import util as suggested, effectively the opposite occours. main.py runs successfully, however now pack.py fails, raising:
Traceback (most recent call last):
File ".../src/pack/pack.py", line 1, in <module>
from . import util
ValueError: Attempted relative import in non-package
I would have thought that imports are relative to the current location, and as such you ought to be able to construct a chain of imports like this. It seems very odd to me that module is supposed to import a sibling file differently depending on where the program starts.
Can someone explain why this error occurs in one way but not the other, and if there is some way to allow this file structure to run whether I want to run from main.py or pack.py?
You will have trouble making the import work in both cases. This is because in one case you are running pack.py as the main file and in another you run it as part of a package.
When you run it as a standalone script python pack.py, the "pack" directory gets added to the PYTHONPATH, meaning that you can import any module in it. Hence, import util will work.
When you run python main.py you add the src directory to your PYTHONPATH. This means that any module or package in src, for example the pack directory, now becomes importable. Hence from pack import pack. However, to access util.py you now need to do from pack import util. You can also do from . import util from within pack.py, as you noticed.
But you can't really do both at the same time. Either src/ is the main directory or src/pack is.
The obvious, but wrong solution, is to let the main.py add the src/pack directory to the PYTHONPATH. That will work, but it's not a good idea. The correct way to do this is to make up your mind. Is src/pack a module that should be imported via import pack or is it just a folder with a bunch of Python scripts? Decide! :-)
I think in this case its' obvious that src/pack should be a module. So then treat it like a module, and make sure it's available like a module. Then you can from pack import util even when running pack.py as a main script.
How do you do that? Well, basically you either install the pack module in your site-packages, or you add the src directory to the PYTHONPATH. That last one is what you want during development. You can either do it manually with export PYTHONPATH=<path> or you can let your testrunners do it for you. You don't have a testrunner? Well you should, but that's another question. :)
For installing it permanently once you don't do development anymore, take a look at Distribute. It includes a testrunner. ;)
Your link is to python 2.7 documentation, but it looks like you are using Python 3.x.
See here: http://docs.python.org/py3k/ for the correct docs.
Python 3 removes implicit relative import. That is, you cannot import packages within the same module in the same way anymore.
You need to use from . import util, this is an explicitly relative import and is allowed. import X no longer checks the current directory. Instead it only check the entries in sys.path. This includes the directory of the script was started and python's standard library.
You are missing __init__.py in your pack directory.
Create an empty file called __init__.py in your pack directory.
Nothing in the correct documentation supports your theory that this form of importing should work.

Categories