We Keep Coding

ModuleNotFoundError on a file in a (working) package when trying to import a class on another folder

Abridged:
When importing a class (in the example below, c2) from another package (folder1), where the imported class (c2) imports a class (c1) from the same package (folder1), the program (file2) raises a ModuleNotFoundError on the import of c1 on c2, even when the import already worked in the package.
Extended:
The example have the following file structure
project/
├── folder1/
│ └── __init__.py
│ └── file1.py
│ └── file2.py
└── folder2/
└── file3.py
where the files in folder1 contain the following classes.
__init__.py is left empty.
(Notice that there's no import error on file2.py)
# file1.py
class c1:
def __init__(self, attr: int = 0):
self.attr = attr
# file2.py
from file1 import c1
class c2:
def __init__(self):
self.attr = [c1() for _ in range(10)]
the file in folder2 imports the class c2
# file3.py
import sys
sys.path.append('../') # to recognize folder1 as a package
from folder1.file2 import c2
but when I try to run file3.py the import of c1 made in file2.py raises ModuleNotFoundError
$ python3 file3.py
Traceback (most recent call last):
File "/home/user/project/folder2/file3.py", line 4, in <module>
from folder1.file2 import c2
File "/home/user/project/folder2/../folder1/file2.py", line 1, in <module>
from file1 import c1
ModuleNotFoundError: No module named 'file1'
Notice that i CAN import c1 on file3.py with the analogous import from folder1.file1 import c1 but can't make it work with c2.
(Of course, this is an abstraction of the actual classes where i found this problem, the actual folder layout is important, but the problem is the same.)
How can I import c2 on folder2/file3.py?
My attempts were trying to import c1 before c2 on file3, also tried to import c1 (and/or c2) in folder1/__init__.py but didn't work, also tried to make folder2/ a package and make the import in its __init__.py but didn't work. Of course i (probably) could simply concatenate file1 and file2 or try to create a package for file1 but i believe I'm doing something wrong on the imports and there must a simple way to solve this.

Add the same correct full path to file1.py in file2.py:
from folder1.file1 import c1
When file2.py trying to import file1.py, it trying to import from ('../') where no file1.py, only /folder1 and /folder2.
And you can delete __init__.py if you are using python 3.3+.

Import multiple subdirectories/modules as if they are just one module

so I have a module/directory called A and it has init.py file and in it, it has another module/directory called B which have its init.py and a file called function.py which has a function called dummy()
here is the structure of directories
A
|-- __init__.py
|
|-- B
|
|-- __init__.py
|-- function.py
so what I want is to be on the same directory that contains directory A and do that
from A import *
dummy()
what I have done is do that in B/init.py
from dummy import *
and that in A/init.py
import B
and I can do that
from A.B import *
I want to write A instead of A.B

I changed your import code a bit and it seems to work now like you wanted.
So in the B directory's init.py it has:
# __init__.py in B
from .function import *
In the A directory's init.py:
# __init__.py in A
from .B import *
Now, when I run Python shell in the directory that contains A and and use from A import *, it calls dummy() with no problem.
However, there are discussions on using wildcard imports in Python. Check this post for example: Should wildcard import be avoided?

How to check does a file imports from another file in Python

Suppose i have a project structure like this
src
└── app
├── main.py
├── db
│ └── database.py
├── models
│ ├── model_a.py
│ └── model_b.py
└── tests
├── test_x.py
└── test_y.py
I want to check which file uses a class or a function from another file. I have a class called Test in main.py
class Test:
pass
I used that class in model_a,
from ..main import Test
But in model_b i used
from ..main import Test
from ..db.database import Data
I want to to check which file uses another file, just like tree command, just a folder name is enough so i tried an old method but it was inefficient ,dirty and that was not something that i expect. The method was i created a file in src named check.py, i imported all packages
from app.db import database
from app.models import model_a, model_b
from app.tests import test_x, test_y
from app import main
print('__file__={0:<35} | __name__={1:<20} | __package__={2:<20}'.format(__file__,__name__,str(__package__)))
And i added this line in the bottom of all files
print('__file__={0:<35} | __name__={1:<20} | __package__={2:<20}'.format(__file__,__name__,str(__package__)))
So when i run check.py i get this result
__file__=/home/yagiz/Desktop/struct/src/app/main.py | __name__=app.main | __package__=app
__file__=/home/yagiz/Desktop/struct/src/app/db/database.py | __name__=app.db.database | __package__=app.db
__file__=/home/yagiz/Desktop/struct/src/app/models/model_a.py | __name__=app.models.model_a | __package__=app.models
__file__=/home/yagiz/Desktop/struct/src/app/models/model_b.py | __name__=app.models.model_b | __package__=app.models
__file__=/home/yagiz/Desktop/struct/src/app/tests/test_x.py | __name__=app.tests.test_x | __package__=app.tests
__file__=/home/yagiz/Desktop/struct/src/app/tests/test_y.py | __name__=app.tests.test_y | __package__=app.tests
__file__=/home/yagiz/Desktop/struct/src/check.py | __name__=__main__ | __package__=None
The result is dirty and doesn't meet my expectations is there a way to get a output like this?
main.py = app/models/model_a, app/models/model_b # These files imports something from main.py
models_b = None # No file imports from models_b
Update, i tried #Hessam Korki's suggestion it doesn't works.
I looked up the source code of modulefinder and i found it adds a badmodule in every import statement which is not useful for me.
Here is how did it go, first i created a function, also i created an another project structure.
src
├── empty.py
├── __init__.py
├── main.py
├── module_finder.py
├── other
│   └── other.py
├── test
│   └── some_app.py
└── this_imports.py
Here is the module_finder.py that contains my function
from modulefinder import ModuleFinder
file_names = ["this_imports.py", "main.py", "test/some_app.py", "other/other.py", "empty.py"]
def check_imports(file_names):
finder = ModuleFinder()
for file in file_names:
finder.run_script(file)
print("\n", file)
for name, mod in finder.modules.items():
print('%s: ' % name, end='')
print(','.join(list(mod.globalnames.keys())[:3]))
print('\n'.join(finder.badmodules.keys()))
Empty file is empty(as expected), in main.py i have
class Test:
pass
In this_imports.py i only have
from src.main import Test
In other/other.py i have
from src.main import Test
from src.test import DifferentTest
And for the last one in test/some_app.py i have
from src.main import Test
class DifferentTest:
pass
So the result should be:
empty.py = None
main.py = None
other/other.py = src.main , src.test
test/some_app.py = src.main
this_imports.py = src.main
But the function gives a wrong result, here is the output:
Filename: this_imports.py
__main__: Test
src.main
Filename: main.py
__main__: Test,__module__,__qualname__
src.main
Filename: test/some_app.py
__main__: Test,__module__,__qualname__
src.main
Filename: other/other.py
__main__: Test,__module__,__qualname__
src.main
src.test
Filename: empty.py
__main__: Test,__module__,__qualname__
src.main
src.test

I believe python's Modulefinder will effectively solve your problem. There is a key named '__main__' in the Modulefinder().items() which holds the modules that were imported in a python file. After running the script through your project and storing the data in a way that suits your purpose, you should be good to go

What you are looking for is to find import dependencies in your package modules. You can run a static analysis on your package directory and parse the import nodes in the syntax trees (ast), and build a dependency graph. Something like below:
import os
from ast import NodeVisitor, parse
import networkx as nx
class Dependency():
def __init__(self, root):
self.root = root
self.base = os.path.basename(root)
self.dependency = nx.DiGraph()
self.visitor = NodeVisitor()
self.visitor.visit_ImportFrom = self.visit_ImportFrom
self.current_node = None
self.dependency.add_node = self.base
def visit_ImportFrom(self, node):
self.dependency.add_edge(node.module, self.current_node)
self.visitor.generic_visit(node)
def run(self):
for root, dirs, files in os.walk(self.root):
for file in files:
full_path = os.path.join(root+os.sep, file)
loc = full_path.split(self.root+os.sep)[1].replace(os.sep,'.')
self.current_node = self.base+'.'+loc
with open(full_path) as fp:
src = fp.read()
tree = parse(src)
self.visitor.generic_visit(tree)
dependency = {}
for src, target in nx.dfs_edges(self.dependency):
if src in dependency:
dependency[src].add(target)
else:
dependency[src] = set([target])
return dependency
For the root location of any package you want to map the import dependencies, you need to do the following then:
root = "path/to/your/src"
d = Dependency(root)
d.run()
This will return the dependency tree (as a dict). Note, we parsed only ImportFrom, you need to add Import to make it complete. Also, all imports are assumed absolute here (i.e. no .. etc). If required, you can add that too (check the level field of the ImportFrom node to do that).

Import a whole folder of python files

I am making a bot in python 3 and wish it to be easily expanded so I have a central file and then one for each command. I wish to know if there is a way to import a sub-directory full of modules without importing each separately. For example:
example
├── commands
│   ├── bar.py
│   └── foo.py
└── main.py
And the code in main.pywould be something like:
import /commands/*
Thanks :D
Solution:
Import each separately with:
from commands import foo, bar
from commands import * Does not work.

If you're using python3, the importlib module can be used to dynamically import modules. On python2.x, there is the __import__ function but I'm not very familiar with the semantics. As a quick example,
I have 2 files in the current directory
# a.py
name = "a"
and
# b.py
name = "b"
In the same directory, I have this
import glob
import importlib
for f in glob.iglob("*.py"):
if f.endswith("load.py"):
continue
mod_name = f.split(".")[0]
print ("importing {}".format(mod_name))
mod = importlib.import_module(mod_name, "")
print ("Imported {}. Name is {}".format(mod, mod.name))
This will print
importing b Imported <module 'b' from '/tmp/x/b.py'>.
Name is b
importing a Imported <module 'a' from '/tmp/x/a.py'>.
Name is a

Import each separately with:
from commands import bar and
from commands import foo
from commands import * Does not work.

How to import classes defined in __init__.py

I am trying to organize some modules for my own use. I have something like this:
lib/
__init__.py
settings.py
foo/
__init__.py
someobject.py
bar/
__init__.py
somethingelse.py
In lib/__init__.py, I want to define some classes to be used if I import lib. However, I can't seem to figure it out without separating the classes into files, and import them in__init__.py.
Rather than say:
lib/
__init__.py
settings.py
helperclass.py
foo/
__init__.py
someobject.py
bar/
__init__.py
somethingelse.py
from lib.settings import Values
from lib.helperclass import Helper
I want something like this:
lib/
__init__.py #Helper defined in this file
settings.py
foo/
__init__.py
someobject.py
bar/
__init__.py
somethingelse.py
from lib.settings import Values
from lib import Helper
Is it possible, or do I have to separate the class into another file?
EDIT
OK, if I import lib from another script, I can access the Helper class. How can I access the Helper class from settings.py?
The example here describes Intra-Package References. I quote "submodules often need to refer to each other". In my case, the lib.settings.py needs the Helper and lib.foo.someobject need access to Helper, so where should I define the Helper class?

'lib/'s parent directory must be in sys.path.
Your 'lib/__init__.py' might look like this:
from . import settings # or just 'import settings' on old Python versions
class Helper(object):
pass
Then the following example should work:
from lib.settings import Values
from lib import Helper
Answer to the edited version of the question:
__init__.py defines how your package looks from outside. If you need to use Helper in settings.py then define Helper in a different file e.g., 'lib/helper.py'.
.
| `-- import_submodule.py
`-- lib
|-- __init__.py
|-- foo
| |-- __init__.py
| `-- someobject.py
|-- helper.py
`-- settings.py
2 directories, 6 files
The command:
$ python import_submodule.py
Output:
settings
helper
Helper in lib.settings
someobject
Helper in lib.foo.someobject
# ./import_submodule.py
import fnmatch, os
from lib.settings import Values
from lib import Helper
print
for root, dirs, files in os.walk('.'):
for f in fnmatch.filter(files, '*.py'):
print "# %s/%s" % (os.path.basename(root), f)
print open(os.path.join(root, f)).read()
print
# lib/helper.py
print 'helper'
class Helper(object):
def __init__(self, module_name):
print "Helper in", module_name
# lib/settings.py
print "settings"
import helper
class Values(object):
pass
helper.Helper(__name__)
# lib/__init__.py
#from __future__ import absolute_import
import settings, foo.someobject, helper
Helper = helper.Helper
# foo/someobject.py
print "someobject"
from .. import helper
helper.Helper(__name__)
# foo/__init__.py
import someobject

If lib/__init__.py defines the Helper class then in settings.py you can use:
from . import Helper
This works because . is the current directory, and acts as a synonym for the lib package from the point of view of the settings module. Note that it is not necessary to export Helper via __all__.
(Confirmed with python 2.7.10, running on Windows.)

You just put them in __init__.py.
So with test/classes.py being:
class A(object): pass
class B(object): pass
... and test/__init__.py being:
from classes import *
class Helper(object): pass
You can import test and have access to A, B and Helper
>>> import test
>>> test.A
<class 'test.classes.A'>
>>> test.B
<class 'test.classes.B'>
>>> test.Helper
<class 'test.Helper'>

Add something like this to lib/__init__.py
from .helperclass import Helper
now you can import it directly:
from lib import Helper

Edit, since i misunderstood the question:
Just put the Helper class in __init__.py. Thats perfectly pythonic. It just feels strange coming from languages like Java.

Yes, it is possible. You might also want to define __all__ in __init__.py files. It's a list of modules that will be imported when you do
from lib import *

Maybe this could work:
import __init__ as lib