This has to be simple, but I swear I've been searching and experimenting for 2 days with no end of errors. The best I've managed to achieve is either the IDE indicates no errors, but fails when running. Or indicates errors, but runs successfully. I'm on Windows 10. Python 3.8.3 conda. I have VS Code, Spyder, and PyCharm--all of which I've uninstalled and reinstalled multiple times hoping one of them would give me useful default behavior.
I think I have a minimally reproducible anomaly that hopefully serves to identify issues, which I'll explain below. As a preface, I'll say that I wish that I could find a complete and direct guide to identifying, declaring, repairing, etc packages and modules. Every source I look at has a different set of suggestions, it seems like. And I'm lost between PYTHONPATH, sys.path, .env workspace files, .vscode json files.
c:\myproject
__init__.py
file1.py
\myenums
__init__.py
category.py
functions.py
general.py
That's leaving out a lot of files and folders. And I won't belabor a long list of variations of errors I've had. Hopefully this pair of examples is telling.
I have a function in \myproject\file1.py and another function in \myproject\myenums\functions.py. Each of those .py files has import references to other .py files. Pretty much all those import statements are red-underlined in VS Code. But depending on how I qualify the import statments--how high up the folder hierarchy I go--one or the other will run but not both. To wit (reduced and simplified indication of actual code):
\myproject\file1.py
import myenums.category # red underline for error
import myenums.functions # red underline for error
import myenums.general # red underline for error
import myenums.results # red underline for error
print(myenums.category) # works
print(myenums.functions) # works
print(myenums.general) # works
print(myenums.results) # works
\myproject\myenums\functions.py
from myenums.category import A, B, C # red underline, note mutual parent folder
from myenums.general import X, Y, Z # red underline, note mutual parent folder
def testfunc(value):
if value is A.x:
return X.a
elif value is A.y:
return Y.a
print(testfunc(A.y)) # doesn't work. ModuleNotFoundError: No module named 'myenums'.
I get the working/not working results by Run Without Debugging in either file.
\myproject\file1.py
import myenums.category # red underline for error
import myenums.functions # red underline for error
import myenums.general # red underline for error
import myenums.results # red underline for error
print(myenums.category) # doesn't work, ModuleNotFoundError: No module named 'category' (on import functions)
print(myenums.functions)
print(myenums.general)
print(myenums.results)
\myproject\myenums\functions.py
from category import A, B, C # red underline, note parent folder qualification removed
from general import X, Y, Z # red underline, note parent folder qualification removed
def testfunc(value):
if value is A.x:
return X.a
elif value is A.y:
return Y.a
print(testfunc(A.y)) # Now works!
So depending on whether I overqualify with the containing folder name in functions.py, I can get functions.py or file1.py to run--but never both. And I can never make error indicators go away on the imports.
I can't figure out at all how to diagnose this problem. Could I have too many or the wrong init.py files? Could they be bad files? They look empty in Notepad. And are not .py.bin. When I run print(p) in sys.path, I do see c:\myproject--but it's a lower case 'c' where all the Anaconda references have upper case 'C'. Hmmm. It's been a long time since I've felt this lost and helpless trying to write some code. I wouldn't have thought that creating references between the files in a single folder would be such a challenge. I have no idea what to try.
Edit: Originally I thought my problem was specific to references between subfolders next to each other in hierarchy, and I thought I would try to fix it by rearranging code to only ever reference to direct subfolders--but this anomaly seems to shoot even that down. Also, I originally posted saying that my anomaly went away. Now I think it's still there, I just had a typo.
Here is my suggestion. Organize your file structure like this for testing purposes:
myproject\
tests.py
myproject\
__init__.py
file1.py
myenums\
__init__.py
category.py
functions.py
general.py
What I did was create a topmost folder for testing purposes that doesn't include a __init__.py file, i.e., it is not part of the main module myproject. You can do all of your testing from here.
The general rule of thumb is that, when creating a Python package, you should not run the scripts directly from inside the package.
To illustrate this, let's write a function inside of myproject/myproject/file1.py:
# myproject/myproject/file1.py
def file1_function():
print("Inside file1.py!")
Now, we have to import it in the topmost __init__.py so that we can access it from outside the function.
# myproject/myproject/__init__.py
from myproject import file1
Established earlier, we are going to run our tests from tests.py, which is outside of the package. In tests.py, we can say:
# myproject/tests.py
import myproject
myproject.file1.file1_function()
Which outputs:
Inside file1.py!
Of course, you could always replace from myproject import file1 with from myproject.file1 import file1_function, which allows us to call file1_function a little more easily:
# myproject/tests.py
import myproject
myproject.file1_function()
Which gives us the same output as before.
What happens when we need to import files/functions/classes/etc. to use elsewhere in the package? Well, we can use absolute imports.
For demonstration purposes, let's say there is an important piece of code in functions.py that file1.py needs to use.
# myproject/myproject/myenums/functions.py
def really_important_function():
print("Inside myproject/myenums/functions.py!")
Now we have to do the same thing that we did above. The only difference is that we are trying to access it from inside the package. Let's move into the __init__.py that is located in myenums:
# myproject/myproject/myenums/__init__.py
from myproject.myenums.functions import really_important_function
Here we use an absolute import, which is an import that stems from the root of the package (myproject is the root).
To use this in file1.py, we then have to import it into file1.py:
# myproject/file1.py
from myenums import really_important_function
def use_important_function():
really_important_function()
Now that we are able to access really_important_function from myproject/myenums/functions.py in myproject/file1.py, we can import it in tests.py the same way we did in the above example:
# myproject/tests.py
import myproject
myproject.file1.use_important_function()
Which outputs:
Inside myproject/myenums/functions.py!
I think it should be worth noting that you don't necessarily need to traverse the package upwards, updating every __init__.py just to test a function that won't be imported directly anyway. If you want to test really_important_function from tests.py, you can do this:
# myproject/tests.py
import myproject
myproject.myenums.functions.really_important_function()
Which gives the correct output.
Conclusion
Another way to look at this is as follows. Let's say you define "function" in functions.py and you want to use it in a function defined in file1.py:
myproject\
tests.py -> import myproject
-> myproject.use_function()
myproject\
__init__.py -> from myproject.file1 import use_function
file1.py -> from myproject.myenums import function
-> def use_function():
function()
myenums\
__init__.py -> from myproject.myenums.functions import function
category.py
functions.py -> def function():
...
general.py
Related
I'm using Python 3.95, and have two files:
shapes.py, which defines class Circle, class Square, and class Triangle
textures.py, which defines class Smooth and class Rough
I'd like to be able to do something like:
from .shapes import *
but I get error ImportError: attempted relative import with no known parent package.
All files are stored in the same folder, and I'm trying to execute Python from that folder, on Python files stored in that folder.
There are numerous posts on SO describing this error, and I've tried their solutions, but to no avail. It seems like some of this depends on Python 2 vs 3, some depends on which folder you're executing from, some depends on packages and modules (which I do not entirely understand).
Some things I've tried which have not worked:
Creating an __init__.py file
Using from shapes import * instead of from .shapes import *
Using from .shapes import Circle
Adding these lines:
import sys
import os
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
sys.path.append(os.path.dirname(SCRIPT_DIR))
This is likely a basic mistake: I'm not trying to build a module or package for distribution, I simply want to put different classes into different files, instead of all my classes in one giant file. How can I do so?
The current directory is only prepended to sys.path automatically if the script is the python file (not if the script is the first argument to python like python app.py).
I think that you can either:
add #! /usr/bin/env python at the start of the main file and make it executable with chmod +x main.py
append the right directoy to sys.path, as it seems to me that you are trying to append the parent dir. SCRIPT_DIR in your last attempt is already the current directory and you don't need to call dirname on it again.
There's no need for a relative import if your main script needs to use a module in the same folder. If you have a main.py and a shapes.py (containing for example a function fun1()), just use any of these:
import shapes
shapes.fun1()
from shapes import fun1
fun1()
from shapes import *
fun1()
Note that the latter is not preferable, since it's not at all clear what you're importing and if shapes.py is later changed to include something that shadows something from an earlier import, it may break your code in surprising ways. Surprise is bad.
If you're writing a package called shapes, create a folder called shapes, put an __init__.py in it, and a Python file which could be called shapes.py or anything you want, as you'll be importing what's relevant from the __init__.py.
Then shapes/__init__.py could be something like:
from .shapes import *
Here, using the * makes a bit more sense, although I'd still favour from .shapes import fun1. The . is not needed, but here it makes sense, because it is relative to another file that's also in the package.
And shapes/shapes.py might be something like:
from .other_mod import fun2
def fun1():
fun2()
Here, the . makes sense if shapes.py and other_mod.py are always going to be sitting next to each other in the package, but perhaps move around within the internal structure. Or perhaps you want to use the explicit . to avoid conflicts with other names.
If other_mod.py is something like:
def fun2():
print('hello')
You could write a main.py in the same folder where the shapes folder sits and the code with shapes imports above would work.
Can you try create new folder and add [shapes.py,textures.py,init.py] files into shapes folder
backend/
|
|------folder_shapes/shapes.py
|------folder_shapes/textures.py
|------folder_shapes/__init__.py
|------your_script.py
|
Then try import from your_script.py
I am working in the following directory tree:
src/
__init__.py
train.py
modules/
__init__.py
encoders/
__init__.py
rnn_encoder.py
My pwd is the top-level directory and my __init__.py files are all empty. I am executing train.py, which contains the following code snippet.
import modules
# RNNEncoder is a class in rnn_encoder.py
encoder = modules.encoders.rnn_encoder.RNNEncoder(**params)
When I execute train.py, I get an error saying that
AttributeError: module 'modules' has no attribute 'encoders'
I am wondering if there is any clean way to make this work. Note that I am not looking for alternative methods of importing, I am well-aware that this can be done in other ways. What I'd like to know is whether it is possible to keep the code in train.py as is while maintaining the given directory structure.
Putting an __init__.py file in a folder allows that folder to act as an import target, even when it's empty. The way you currently have things set up, the following should work:
from modules.encoders import rnn_encoder
encoder = rnn_encoder.RNNEncoder(**params)
Here, python treats modules.encoders as a filepath, essentially, and then tries to actually import the code inside rnn_encoder.
However, this won't work:
import modules
encoder = modules.encoders.rnn_encoder.RNNEncoder(**params)
The reason is that, when you do import modules, what python is doing behind the scenes is importing __init__.py from the modules folder, and nothing else. It doesn't run into an error, since __init__.py exists, but since it's empty it doesn't actually do much of anything.
You can put code in __init__.py to fill out your module's namespace and allow people to access that namespace from outside your module. To solve your problem, make the following changes:
modules/encoders/__init__.py
from . import rnn_encoder
modules/__init__.py
from . import encoders
This imports rnn_encoder and assigns it to the namespace of encoders, allowing you to import encoders and then access encoders.rnn_encoder. Same with modules.encoders, except a step upwards.
i've run through many posts about this, but still doesn't seem to work. The deal is pretty cut. I've the got the following hierarchy.
main.py
DirA/
__init__.py
hello.py
DirB/
__init__.py
foo.py
bla.py
lol.py
The__init__.py at DirA is empty. The respective one at DirB just contains the foo module.
__all__.py = ["foo"]
The main.py has the following code
import DirA
import DirB
hey() #Def written at hello.py
foolish1() #Def written at foo.py
foolish2() #Def written at foo.py
Long story short, I got NameError: name 'foo' is not defined. Any ideas? Thanks in advance.
You only get what you import. Therefore, in you main, you only get DirA and DirB. You would use them in one of those ways:
import DirA
DirA.something_in_init_py()
# Importing hello:
import DirA.hello
DirA.hello.something_in_hello_py()
# Using a named import:
from DirA.hello import something_in_hello_py
something_in_hello_py()
And in DirB, just make the __init__.py empty as well. The only use of __all__ is for when you want to import *, which you don't want because, as they say, explicit is better than implicit.
But in case you are curious, it would work this way:
from DirB import *
something_in_dirb()
By default the import * will import everything it can find that does not start with an underscore. Specifying a __all__ restricts what it imported to the names defined in __all__. See this question for more details.
Edit: about init.
The __init__.py is not really connected to the importing stuff. It is just a special file with the following properties:
Its existence means the directory is a python package, with several modules in it. If it does not exist, python will refuse to import anything from the directory.
It will always be loaded before loading anything else in the directory.
Its content will be available as the package itself.
Just try it put this in DirA/__init__.py:
foo = 42
Now, in your main:
from DirA import foo
print(foo) # 42
It can be useful, because you can import some of your submodules in the __init__.py to hide the inner structure of your package. Suppose you build an application with classes Author, Book and Review. To make it easier to read, you give each class its own file in a package. Now in your main, you have to import the full path:
from myapp.author import Author
from myapp.book import Book
from myapp.review import Review
Clearly not optimal. Now suppose you put those exact lines above in your __init__.py, you may simplify you main like this:
from myapp import Author, Book, Review
Python will load the __init__.py, which will in turn load all submodules and import the classes, making them available on the package. Now your main does not need to know where the classes are actually implemented.
Have you tried something like this:
One way
from DirA import hello
Another way
from DirA.hello import hey
If those don't work then append a new system path
You need to import the function itself:
How to call a function from another file in Python?
In your case:
from DirA import foolish1, foolish2
In the below hierachy, is there a convenient and universal way to reference to the top_package using a generic term in all .py file below? I would like to have a consistent way to import other modules, so that even when the "top_package" changes name nothing breaks.
I am not in favour of using the relative import like "..level_one_a" as relative path will be different to each python file below. I am looking for a way that:
Each python file can have the same import statement for the same module in the package.
A decoupling reference to "top_package" in any .py file inside the package, so whatever name "top_package" changes to, nothing breaks.
top_package/
__init__.py
level_one_a/
__init__.py
my_lib.py
level_two/
__init__.py
hello_world.py
level_one_b/
__init__.py
my_lib.py
main.py
This should do the job:
top_package = __import__(__name__.split('.')[0])
The trick here is that for every module the __name__ variable contains the full path to the module separated by dots such as, for example, top_package.level_one_a.my_lib. Hence, if you want to get the top package name, you just need to get the first component of the path and import it using __import__.
Despite the variable name used to access the package is still called top_package, you can rename the package and if will still work.
Put your package and the main script into an outer container directory, like this:
container/
main.py
top_package/
__init__.py
level_one_a/
__init__.py
my_lib.py
level_two/
__init__.py
hello_world.py
level_one_b/
__init__.py
my_lib.py
When main.py is run, its parent directory (container) will be automatically added to the start of sys.path. And since top_package is now in the same directory, it can be imported from anywhere within the package tree.
So hello_world.py could import level_one_b/my_lib.py like this:
from top_package.level_one_b import my_lib
No matter what the name of the container directory is, or where it is located, the imports will always work with this arrangement.
But note that, in your original example, top_package it could easily function as the container directory itself. All you would have to do is remove top_package/__init__.py, and you would be left with efectively the same arrangement.
The previous import statement would then change to:
from level_one_b import my_lib
and you would be free to rename top_package however you wished.
You could use a combination of the __import__() function and the __path__ attribute of a package.
For example, suppose you wish to import <whatever>.level_one_a.level_two.hello_world from somewhere else in the package. You could do something like this:
import os
_temp = __import__(__path__[0].split(os.sep)[0] + ".level_one_a.level_two.hello_world")
my_hello_world = _temp.level_one_a.level_two.hello_world
This code is independent of the name of the top level package and can be used anywhere in the package. It's also pretty ugly.
This works from within a library module:
import __main__ as main_package
TOP_PACKAGE = main_package.__package__.split('.')[0]
I believe #2 is impossible without using relative imports or the named package. You have to specify what module to import either by explicitly calling its name or using a relative import. otherwise how would the interpreter know what you want?
If you make your application launcher one level above top_level/ and have it import top_level you can then reference top_level.* from anywhere inside the top_level package.
(I can show you an example from software I'm working on: http://github.com/toddself/beerlog/)
I'm taking a look at how the model system in django works and I noticed something that I don't understand.
I know that you create an empty __init__.py file to specify that the current directory is a package. And that you can set some variable in __init__.py so that import * works properly.
But django adds a bunch of from ... import ... statements and defines a bunch of classes in __init__.py. Why? Doesn't this just make things look messy? Is there a reason that requires this code in __init__.py?
All imports in __init__.py are made available when you import the package (directory) that contains it.
Example:
./dir/__init__.py:
import something
./test.py:
import dir
# can now use dir.something
EDIT: forgot to mention, the code in __init__.py runs the first time you import any module from that directory. So it's normally a good place to put any package-level initialisation code.
EDIT2: dgrant pointed out to a possible confusion in my example. In __init__.py import something can import any module, not necessary from the package. For example, we can replace it with import datetime, then in our top level test.py both of these snippets will work:
import dir
print dir.datetime.datetime.now()
and
import dir.some_module_in_dir
print dir.datetime.datetime.now()
The bottom line is: all names assigned in __init__.py, be it imported modules, functions or classes, are automatically available in the package namespace whenever you import the package or a module in the package.
It's just personal preference really, and has to do with the layout of your python modules.
Let's say you have a module called erikutils. There are two ways that it can be a module, either you have a file called erikutils.py on your sys.path or you have a directory called erikutils on your sys.path with an empty __init__.py file inside it. Then let's say you have a bunch of modules called fileutils, procutils, parseutils and you want those to be sub-modules under erikutils. So you make some .py files called fileutils.py, procutils.py, and parseutils.py:
erikutils
__init__.py
fileutils.py
procutils.py
parseutils.py
Maybe you have a few functions that just don't belong in the fileutils, procutils, or parseutils modules. And let's say you don't feel like creating a new module called miscutils. AND, you'd like to be able to call the function like so:
erikutils.foo()
erikutils.bar()
rather than doing
erikutils.miscutils.foo()
erikutils.miscutils.bar()
So because the erikutils module is a directory, not a file, we have to define it's functions inside the __init__.py file.
In django, the best example I can think of is django.db.models.fields. ALL the django *Field classes are defined in the __init__.py file in the django/db/models/fields directory. I guess they did this because they didn't want to cram everything into a hypothetical django/db/models/fields.py model, so they split it out into a few submodules (related.py, files.py, for example) and they stuck the made *Field definitions in the fields module itself (hence, __init__.py).
Using the __init__.py file allows you to make the internal package structure invisible from the outside. If the internal structure changes (e.g. because you split one fat module into two) you only have to adjust the __init__.py file, but not the code that depends on the package. You can also make parts of your package invisible, e.g. if they are not ready for general usage.
Note that you can use the del command, so a typical __init__.py may look like this:
from somemodule import some_function1, some_function2, SomeObject
del somemodule
Now if you decide to split somemodule the new __init__.py might be:
from somemodule1 import some_function1, some_function2
from somemodule2 import SomeObject
del somemodule1
del somemodule2
From the outside the package still looks exactly as before.
"We recommend not putting much code in an __init__.py file, though. Programmers do not expect actual logic to happen in this file, and much like with from x import *, it can trip them up if they are looking for the declaration of a particular piece of code and can't find it until they check __init__.py. "
-- Python Object-Oriented Programming Fourth Edition Steven F. Lott Dusty Phillips