I'm upgrading a bunch of scripts where the ecosystem is a bit of a mess. The scripts always relied on external modules, and didn't have any package infrastructure of their own (they also didn't do much OOP, as you can imagine). There's nothing at the top level, but it is the working directory when starting Python and I'd like to keep it that way. At the top-level, I've just created __init__.py file (based on another question). As I'm less experienced with Python __init__.py confuses me a bit. All of the __init__.py files I've created are empty, it's my understanding that this is all that's required.
Assume I have the following directory structure:
__init__.py
dev.properties
prod.properties
F/
Foo.py
__init__.py
B/
bar.py
__init__.py
And the code is like this :
# Foo.py
from ..b import bar
barFunc()
# bar.py
def barFunc():
print "Hello, World!"
sys.stdout.flush()
I've created __init__.py at the root, in F/ and in B/. However, when I run python F/Foo.py, I get an error:
Traceback (most recent call last):
File "F/Foo.py", line 3, in <module>
from ..b import bar
ValueError: Attempted relative import in non-package
What exactly would I need to do to invoke python F/Foo.py and be able to depend on things defined in sibling directories?
Update
Thanks to #user2455127, I realized that I forgot to remove the file extension .py and my working directory was wrong. From the mypackage directory, running python -m mypackage/F/Foo, the error was : myvirtualenv/bin/python: No module named mypackage/B/bar.
Re-reading #user2455127's post, I ran from the directory above and get a long Traceback:
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "<full path>/mypackage/foo/Foo.py", line 24, in <module>
from ..b import bar
ValueError: Attempted relative import in non-package
I'm not quite sure what needs to be done to fix this, but it seems like the __package__ attribute may help. I'll try and figure that out, and post another update.
If the current working directory is F's and B's parent directory, then F and B are available as modules to all Python code. You should run:
$ F/foo.py
and then F/foo.py should contain
from B.bar import barFunc
barFunc()
As for __init__.py, that file's existance simply makes the directory an importable module. If you want to know more about it, check the docs on how imports work. (For most people, reading all of that isn't necessary.)
Relative imports can be pretty messy, so I'd advise shying away from them for now.
Have a look to this :
Attempted relative import in non-package even with init.py and brenBarn's answer
If you run it from the folder upper than the one with dev.properties and the others files (called lambda in my case), with this command line :
python -m lambda.F.Foo
it works.
Related
__init__.py files are meant to initialize packages and/or to set what directory is a package, as far as I know.
So my question is, even though these are implicitly run when a package/module is imported, is it normal to receive errors when you run that __init__.py file directly? and why?
I was testing a project I'm currently working on, which is working great, no errors when I import from root files, but what got my attention is that when I execute every module just to check for errors, all the __init__.py files are the ones that throws errors.
For the sake of education,
I followed these 'tutorials' and ran every __init__.py files of every package (directly, not importing a module/package) just to identify if I'm doing something wrong and received errors on those too.
Please, take a look at these examples:
Article: Understanding Python imports, __init__.py and pythonpath — once and for all
Traceback (most recent call last):
File "c:\Users\username\Desktop\python_packages_tutorial\utils\__init__.py", line 1, in <module>
from utils.lower import to_lower
ModuleNotFoundError: No module named 'utils'
I even added the directory to PYTHONPATH env variable, and sys.path even recognizes the directory where it is located. As I said before, it's just happening when I directly execute the init.py file, the rest is working well.
Article: Introduction to Create Own Python Packages
Traceback (most recent call last):
File "c:\Users\username\Desktop\medium_python_packages\my_package\__init__.py", line 1, in <module>
from .my_module1 import *
^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: attempted relative import with no known parent package
My project __init__.py file:
sys.path
[
'c:\\Users\\username\\Desktop\\code\\projects\\flask_app\\register',
'C:\\Users\\username\\Desktop\\code\\projects\\flask_app\\register', <-- PYTHONPATH env
'C:\\Users\\username\\AppData\\Local\\Programs\\Python\\Python311\\python311.zip',
'C:\\Users\\username\\AppData\\Local\\Programs\\Python\\Python311\\DLLs',
'C:\\Users\\username\\AppData\\Local\\Programs\\Python\\Python311\\Lib',
'C:\\Users\\username\\AppData\\Local\\Programs\\Python\\Python311',
'c:\\Users\\username\\Desktop\\code\\projects\\flask_app\\flask',
'c:\\Users\\username\\Desktop\\code\\projects\\flask_app\\flask\\Lib\\site-packages'
]
Traceback
Traceback (most recent call last):
File "c:\Users\username\Desktop\code\projects\flask_app\register\__init__.py", line 10, in <module>
from register.routes import *
ModuleNotFoundError: No module named 'register'
But they are working as expected.
I have the following file/folder structure:
testpackage (folder)
src (folder)
__init__.py
module1.py
tests
__init__.py
test_module1.py
just for clearance: the "module1.py" is under the "src" folder which is under the "testpakcage" folder.
"tests" is also under the "testpakcage" folder - same level as the "src" one.
module1.py has a class named "class1" as so:
class class1:
def method1 (self):
print('i am method1')
in test_module1.py I want to run tests on the above module. this is it's contents:
import unittest
from testpackage.src import module1
t = module1.class1()
t.method1()
this package is not installed, and I don't instead to install or submit it anywereh, I'm just trying to find the best structuring practice for me, for future packaging creation.
problem is: when I run the following either from the "tests" or "testpackage" folder:
/usr/bin/python3.6 -m unittest discover
I get the following error:
E
======================================================================
ERROR: test_module1 (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_module1
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib64/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
something similar also happens when I just try to run "test_module1.py" from the "tests" folder:
Traceback (most recent call last):
File "test_module1.py", line 5, in <module>
from testpackage.src import *
ModuleNotFoundError: No module named 'testpackage'
so I tried changing the "import" line with a few alternatives but none of them work. each one of those was a different attempt (not all of them at once):
from testpackage.src import *
import testpackage.src.module1 as module1
import ..src.module1
from ..src.module1 import class1
searching stackoverflow I found solutions that worked for some but not for those using python 3 and above - which is my case.
any suggestions? I think what I'm trying to do is rather simple and I'm missing something really basic here.
I'm using python3.6 by way
I'm wondering if you saw different errors for some of those different import attempts you made. Not sure if this will solve your problem, but this is one way it would generally be accomplished.
First, I am not sure what the top level of your package is supposed to be. Do you reference testpackage in any of the code in the src folder? If so, that top folder should also contain an __init__.py file.
Now, at the very top of your test_module1.py file, add:
import sys
pkg_dir = ".." # if "src" if the top level folder else "..." if testpackage is the top level folder
sys.path.append(pkg_dir)
Note you must change pkg_dir depending on your module's structure.(which I cannot tell from your question).
What this code does is add the folder containing the top level folder of your package to the python import search tree. This can solve problems that the relative import in your example file will not: if the files in your module use module-level imports (e.g. import testpackage.src.module2 in module1.py). This is common in packages with multiple submodules that cross-import.
I created a python project, using Pycharm if it matters, that looks like this:
outer_dir/
outside_main.py
module_example/
module_main.py
tests/
test_1.py
…
level_1/
foo_1.py
level_2/
foo_2.py
main.py calls functions from foo_1.py
foo_1.py calls functions from foo_2.py
The general use case I'm trying to emulate is a python "package" or repo called module_example that has been downloaded and used by outside_main.py at the same level as the top-level package directory.
Here's how module_main.py looks:
# module_main.py
from level_1.foo_1 import foo_function_1
from level_1.level_2.foo_2 import foo_function_2
def main():
print(foo_1())
print(foo_2())
if __name__ == "__main__":
main()
And I can call this top level script from test_1.py just fine like this:
# test_1.py
from module_main import main
if __name__ == "__main__":
main()
However, when I try to use main() from outside the module_example directory, with code like this:
# outside_main.py
from module_example.module_main import main
if __name__ == "__main__":
main()
I get this error:
Traceback (most recent call last):
File "/path/to/outside_main.py", line 1, in <module>
from module_example.module_main import main
File "/path/to/module_example/module_main.py", line 1, in <module>
from level_1.foo_1 import foo_function_1
ModuleNotFoundError: No module named 'level_1'
What seems to be happening is that regardless of directory of the python script being imported, its own imports are being searched for relative to the directory from which the main script is executed.
I'm looking for a method of importing that works equally whether I'm calling outside_main.py or test_1.py, and regardless of what directory the terminal is currently pointing at.
I've read this post about how python imports work, and it basically suggests I'm out of luck, but that makes no sense to me, all I'm trying to do is emulate using someone's open source library from github, they usually have a tests folder within their repo, that works fine, but also I can just unzip the entire repo in my project and call import on their py files without issue. How do they do it?
Before entering the hell of sys.path-patching, I would suggest to wrap your code that currently lives in module_example into a package_example directory, add a setup.py and throw a number of __init__.py in.
Make package_example a git repo.
In every other project where you want to use module_example.[...], you create a venv and pip -e /path/to/package_example.
outer_dir/
outside_main.py
package_example/
.git/
...
setup.py
module_example/
__init__.py
module_main.py
tests/
...
where __init__.py (one in each subdirectory that contains modules that are to be imported) can just be empty files and setup.py has to be filled according to the distutils documentation.
Solution 1:
Use explicit relative imports (relative to the files inside the directory module_example). Note that the usage of implicit relative imports has already been removed in Python3
Implicit relative imports should never be used and have been removed in Python 3.
outer_dir/module_example/module_main.py
# Note the "." in the beginning signifying it is located in the current folder (use ".." to go to parent folder)
from .level_1.foo_1 import foo_function_1
from .level_1.level_2.foo_2 import foo_function_2
# All the rest the same
Solution 2:
Include the path to module_example in your PYTHONPATH environment variable (or you can also choose to update sys.path directly)
export PYTHONPATH=$(pwd):$(pwd)/module_example
Output:
Both solutions above was successful for me.
Before the fix:
$ python3 outside_main.py
Traceback (most recent call last):
File "outside_main.py", line 1, in <module>
from module_example.module_main import main
File "/home/nponcian/Documents/PearlPay/Program/2021_07Jul_31_StackOverflow/1/module_example/module_main.py", line 1, in <module>
from level_1.foo_1 import foo_function_1
ModuleNotFoundError: No module named 'level_1'
After the fix:
$ python3 outside_main.py
This is foo_function_1
This is foo_function_2
Suppose I have the following simple project structure.
project/
package/
__init__.py
__main__.py
module.py
requirements.txt
README.md
After doing some research on Google, I've tried to make it reflect general best practices for a very simple console application (but not as simple as simply having a single script). Suppose __init__.py contains simply print("Hello from __init__.py") and module.py contains a similar statement.
How should I do imports inside of __main__.py, and then how should I run my project?
Let's say first that __main__.py looks simply like this:
import module
print("Hello from __main__.py")
If I run my project with the simple command python package, I get this output:
Hello from module.py
Hello from __main__.py
As can be seen, __init__.py didn't run. I think this is because I'm running my project's package as a script. If I instead run it as a module with the command python -m package I get this output:
Hello from __init__.py
Traceback (most recent call last):
File "C:\Users\MY_USER\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\MY_USER\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\MY_USER\PATH\TO\PROJECT\project\package\__main__.py", line 1, in <module>
import module
ModuleNotFoundError: No module named 'module'
If I change the first line in __main__.py to import package.module and run python -m package again I get this output:
Hello from __init__.py
Hello from module.py
Hello from __main__.py
Great! It seems everything runs properly now when running my project's package as a module. Now what if I try python package again and run it as a script?
Traceback (most recent call last):
File "C:\Users\MY_USER\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\MY_USER\AppData\Local\Programs\Python\Python38\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "package\__main__.py", line 1, in <module>
import package.module
ModuleNotFoundError: No module named 'package'
Alright. So please correct me if I'm wrong, but it seems that I have two options. I can either write the imports in my package to work with running it as a script or running as a module, but not both. Which is better, if one is indeed preferable, and why? When would you use the command python package vs python -m package, and why? Is there some general rule for writing imports within a simple project that I might not be understanding? Am I missing something else fundamental?
In summary: What is the best practice in this situation, why is it the best practice, and when would you set your project up for the alternative approach (python package vs python -m package)?
The most common way to distribute executable python code is to package it into an installable .wheel file. If it's only a single file, you can also just distribute that. But as soon as you got two files, you run into the exact import issues that you experience, at which point you need some metadata to have a well-defined and visible toplevel for imports (which would e.g. be package.module in your case), entrypoints for script code, third party dependencies... all of which is achieved by "making the code installable".
If you like technical documentation, you can read up on what exactly that means by going through this and this tutorial by the python packaging authority (PyPA).
To get you started with your project though, what you are missing is a setup.py file which will contain install instructions and a script entrypoint to provide a way to run executable code from within your package:
from setuptools import setup
with open("requirements.txt") as f:
requirements = [line.strip() for line in f.readlines()]
setup(
# obligatory, just the name of the package. you called it "package" in your
# example so that's the name here
name="package",
# obligatory, when it's done you can give it a 1.0
version="0.1",
# point the installer to the module (read, folder) that contains your code,
# by convention usually the same as the package name
packages=["package"],
# if there are dependencies, specify them here. actually you can delete the
# requirements.txt and just paste the content here, but this here will also work
install_requires=requirements,
# point the installer to the function that will run your executable code.
# the key name has got to be 'console_script', the value content is up
# to you, with its interpretation being:
# 'package_command' -> the name that you can call your code by
# 'package.__main__' -> the path to the file that you want to call
# 'test' -> the actual function that contains the code
entry_points={'console_scripts': ['package_command=package.__main__:test']}
)
After adding this file as project/setup.py, you need to keep the following things in mind:
put your script code (e.g. print('hello world)') into a function called test within your __main__.py file[1]
run pip install -e . to install your package locally[2]
execute the script code by running package_command on the command line - neither python package nor python -m package is best practice to run installable python scripts
[1] Binding a simple function to a script-entrypoint is as basic as it gets. You probably don't want to re-invent things like help-texts, argument parsing/verifying, ... so if you really mean to write a cli-application, you might want to look into something like click to handle the tedious stuff for you.
[2] This performs a dev-install, which is convenient during development since it means that you can test behavior as you work on it. If you want to distribute your code, you'd run pip wheel . (might require running pip install wheel before). This will create a wheel file that you can give to your clients so they can install it manually with pip install package-0.1-py3-none-any.whl, after which package_command would also be available on their systems.
Why am I getting import error for a module I have in the project. All the packages are under the project, they all have __init __.py and other scripts do not give the same error. Python version is 3.6. Code was written in Unix environment.
Here is the import error I get. I am trying to run a test here.
Ran 1 test in 0.001s
FAILED (errors=1)
Error
Traceback (most recent call last):
File "/usr/lib/python3.6/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/lib/python3.6/unittest/case.py", line 605, in run
testMethod()
File "/usr/lib/python3.6/unittest/loader.py", line 34, in testFailure
raise self._exception
ImportError: Failed to import test module: test_SMSHandler
Traceback (most recent call last):
File "/usr/lib/python3.6/unittest/loader.py", line 153, in loadTestsFromName
module = __import__(module_name)
File "/home/sevvalboylu/server/app/api/test_SMSHandler.py", line 11, in <module>
from middleware.services import Sender
ModuleNotFoundError: No module named 'middleware'
Looks like you are missing a project's root path in PYTHONPATH
From the docs (Modules - The Module Source Path)
When a module named spam is imported, the interpreter first searches
for a built-in module with that name. If not found, it then searches
for a file named spam.py in a list of directories given by the
variable sys.path. sys.path is initialized from these locations:
The directory containing the input script (or the current directory
when no file is specified).
PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH).
The installation-dependent default.
If this solution doesn't work for you, please post the project's tree to make it easier find the problem.
I've experienced a similar problem with import error when running unit tests (with correct importable structure), but the cause and the solution are different to described in the answer above. Still this is relevant and may help somebody.
In my case I have structure like that (__init__.py present but omitted below for brevity):
mypkg
\-- foo.py
another
tests
\-- some
\-- <tests here>
mypkg <--- same name as top level module mypkg
\-- test_a.py
When running from top level directory imports in test_a.py of modules from mypkg were failing:
# test_a.py
from mypkg import foo # causes ModuleNotFoundError: No module named 'mypkg.foo'
Renaming mypkg folder under tests folder into something else solved the problem.