We Keep Coding

Cannot run Python3 script from subfolder // relative import error [duplicate]

This question already has answers here:
How to do relative imports in Python?
(18 answers)
Closed 4 years ago.
I have a simple project structure like this:
➜ (venv:evernote) evernote_bear_project git:(master) ✗ tree | grep -v pyc
.
├── README.md
...
(snip)
...
├── manage.py
├── sample
│ ├── EDAMTest.py <==== here is an import that won't work
│ └── enlogo.png
└── util
├── __init__.py
├── files.py <====== This is being imported
└── test_files.py
Now I have a relative import in sample/EDAMTest.py:
from ..util.files import *
When I try to run python sample/EDAMTest.py from project root folder in command line, I get an error saying:
ValueError: attempted relative import beyond top-level package
I know this has been asked many times, but I still don't get it.
Since I'm running the script from the project root, in my understanding Python should be able to "know" that when I try to import from ..util.files import *, that it should go up one directory, no?
EDIT
Thanks for all the answers.
So what I understand from the link above is this:
I was running the module sample/EDAMTest.py directly via python sample/EDAMTest.py, and that meant
that the __name__ of the module was __main__
and now the path (sample/) was my "root" so to speak.
So now Python searches only this path and any path that's below it for modules / packages. Hence the error message attempted relative import _beyond top-level package_, so it cannot go one more level up, because it is at the root already.
Also Python cannot look one level "up", since this syntax from ..util.files import * does not go up a level in directory, but in a list of modules / packages it keeps on the "import search path" (sys.path)?
Is that correct?

sys.path.append() is a tweak, if your directory structure is fixed and there is nothing you can do about it.
Otherwise, you can try rearranging the folders. The easiest is moving util under sample, another option is making both of the folders psrt of a larger package.
Also import * is not ensorsed.

The relative import syntax is for importing other modules from the same package, not from the file system.
Depending on what you want, you could...
Fix the package so that the relative import works
Put __init__.py files in the project root and sample directory and run the script from one level up. This doesn't seem like what you want.
Tell python where to find the package
Set the PYTHONPATH environment variable so that python can find the package.
PYTHONPATH='.':$PYTHONPATH python samples/EDAMTest.py
Install util so that python can find it
Add a setup script and use it to install the util package and avoid setting PYTHONPATH.

"The relative import syntax is for importing other modules from the same package, not from the file system.", This is right as stated by George G.
Put __init__.py in your subfolders, which will make them package.
__init__.py can be an empty file but it is often used to perform
setup needed for the package(import things, load things into path, etc).
However you can import File into your __init__.py to make it
available at the package level:
# in your __init__.py
from util import files
# now import File from util package
from util import files
of if you want to import some specific method or class, you can do
from util.files import some_function
Another thing to do is at the package level make util/modules
available with the __all__ variable. When the interpeter sees
an __all__ variable defined in an __init__.py it imports the
modules listed in the __all__ variable when you do:
from package import *
__all__ is a list containing the names of modules that you want to be
imported with import * so looking at our above example again if we
wanted to import the submodules in util the all variable
in util/init.py would be:
__all__ = ['files', 'test_files']
With the __all__ variable populated like that, when you perform
from util import *
it would import files and test_files.

Python3.5 import from subdirectory doesn't work

I have a bunch of python scripts and simply want to structure them by putting most of them into subdirectories. However, when I try to load scripts from subdirectories, python gives me different error messages, depending on how I try to import the subdirectory scripts.
My subdir looks like this:
io
├── dataset_creator.py
└── read_data.py
In my script from the parent dir, when I do
from io import dataset_creator
this error occurs:
ImportError: cannot import name 'dataset_creator'
When I do
import io.dataset_creator
this error occurs:
ImportError: No module named 'io.dataset_creator'; 'io' is not a package
I also touched __init__.py into io/ but it didn't help at all, as well as preceeding a dot to io/, but no luck. The python docs say I should add the __init__.py and then everything should work, basically (as far as I interpreted it).
Can anyone help me here? If I left out some important info, please tell me and I'll add it.
Cheers,
Jakob
EDIT:
As many of you stated, io is already another package in python, so renaming my io/ to something different fixed the problem (while also having the __init__.py). Thank you very much!
I know there have been multiple correct answers, however, I could just mark one as correct, sorry.

The name io is already being used by a standard library module. Since it's one of the very basic modules used by the interpreter, it gets loaded during the startup process, before any of your code runs. This means that by the time Python sees your request to import io.dataset_creator, it's already got an io module in sys.modules. Since that module is not a package, it won't try loading the other submodule you've written in your io package (even if you had a module search path set up so that your package came ahead of the standard library).
You should rename your io package. One option is to put it inside another package (mypackage.io.dataset_creator should work fine). You could also just replace the name io with something more specific (e.g. myproject_io).

It's possible that that it's failing because io is already a built-in module

I have answered a similar question here Using exec on a file in a different directory causes module import errors
Append your parent path to Pythonpath:
import sys
sys.path.append("/path/to/parentfolder")
You can use os.path.dirname(__file__) to get file's absolute path other than hardcoded path.
Add __init__.py to your parent folder and io folder, make the it as python package other than directory.
import the module:
import io.dataset_creator as dcreator
parent/
-- app.py
-- io/
--dataset_creator.py
--read_data.py
In you app.py:
import sys
sys.path.append(os.path.abspath(os.path.dirname(__file__)))
import io.dataset_creator as dcreator

This happened to me as well on Python 3.5.1 when I tested it.
Renaming the directory io to something else (I used my_io) fixed the problem. Here was my test case:
main.py
my_io
├── module.py
└── something.py
Both modules imported correctly when I changed the directory's name. I suggest you change your io directory to something similar to avoid this.
I think this must be to do with some internal Python module called io which was conflicting somehow.

Recreating the problem:
mkdir io
touch dataset_creator.py
touch read_data.py
python3 -c 'from io import dataset_creator'
python3 -c 'import io.dataset_creator'
Gives the error messages.
Solution:
Create another sub-directory called "io" and put the files there.
Use a name different than "io", as a module by that name already exists.
Explanation:
You are already in the io dir, so you don't need to specify the "io". You can simply do:
python3 -c 'import dataset_creator'
python3 -c 'import read_data'
And once you add a function or class in your python files:
def hello_world():
print("hello world")
You can import like this:
python3 -c 'from read_data import hello_world'
To organise your code under an io module umbrella, create another io directory as follows and use it to store your python code:
ia (parent dir where you do the import)
├── ia
│ ├── dataset_creator.py
│ └── read_data.py
├── .gitignore
├── requirements.txt
├── setup.py
└── README.md
python3 -c 'import ia.dataset_creator'
Note I renamed the directory to "ia" as well as there is already an "io" module that exists (ref).

relative import in Python 3

Say my project is structured like this:
myproject
├── calendar.py
├── foo.py
└── __init__.py
In foo.py, I have
from calendar import isleap
I thought in Python 3.x, without using the explicit .calendar the code above should load the build-in calendar module instead my own calendar module, but apparently my local calendar.py is still being imported and it throws an error because there's no 'isleap' in mypkg/calendar.py. Why was my local calendar module imported here?
I had to rename calendar.py to cal.py to get this work..

from __future__ import absolute_import is the default on Python 3. Therefore from calendar import isleap statement imports the top-level module calendar.
If you see other results; it means either you are not using Python 3 or you are trying to run a python module from inside a package as a script (myproject directory itself is in sys.path). If the latter then your calendar.py becomes the top-level module and (due to the current directory comes before stdlib directories in sys.path) from calendar import isleap imports calendar.py from the current directory. "Never add a package directory, or any directory inside a package, directly to the Python path"
To avoid it, do not run modules from within python packages directly e.g., do not do this: cd myproject; python foo.py. Do this instead: python -mmyproject.foo (or you could define what scripts should be run in setup.py or create a similar script manually: from myproject import foo; foo.main()).
If you want to run a Python package as a script then create myproject/__main__.py then run python -mmyproject.
If you want to do a relative import in Python 3; do it explicitly e.g., in myproject/foo.py:
from .calendar import something
Or do an absolute import:
from myproject.calendar import something

It looks like your path or directory structure is set up wrong.
Given the following structure the full name of your calendar module should be myproject.calendar. You can check this by printing out the __name__ attribute of your module. For this to be the case, the path that your program uses to import local modules must be the folder containing myproject.
myproject
├── calendar.py
├── foo.py
└── __init__.py
It seems like the path you are using is actually myproject. Meaning calendar.py is turned into the root level module calendar, rather than myproject.calendar. Python prefers local modules to builtin ones, and so imports your calendar module.
More typically you might do something like this.
MyProjectFolder
├── main.py
└── myproject
├── calendar.py
├── foo.py
└── __init__.py
And then run your program like this:
#! /bin/bash
cd /path/to/MyProjectFolder
python main.py

Python will check your local modules and load them firstly by import.
from calendar import isleap will search the module calendar in your locale package firstly. If not found, it will import from the builtin library calendar.
from .calendar import isleap will only import from your locale module calendar. If not found, raises a exception ImportError.
That is why you should use relative import in a package.
You can do a trick like that to import the builtin library without checking of the local modules. But it's only a trick. I will never use it in production. You should better rename your module calendar.
import imp, sys
f, pathname, desc = imp.find_module("calendar", sys.path[1:])
calendar = imp.load_module("calendar", f, pathname, desc)
f.close()
from calendar import isleap

How did the import statement find that module?

Given the following directory structure:
here/
├── app
│   ├── __init__.py
│   ├── json.py
│   └── example.py
└── my_script.py
__init__.py and json.py are empty files.
Contents of my_script.py:
from app import example
Contents of example.py
import importlib, imp, sys, os
# ensures '' is not in sys.path
sys.path = [p for p in sys.path if p]
# ensures PYTHONPATH, if any, is not over-reaching
os.environ.pop('PYTHONPATH', None)
# ensures we do not see json.py in the cwd
assert not os.path.isfile('json.py')
print '1: ', imp.find_module('json')
print '2: ', __import__('json')
print '3: ', importlib.import_module('json')
import json
json.loads
Now, from the here directory, execute:
python ./my_script.py
You will see that methods 1, 2, 3, all find the core library version of json module.
However, the actual import statement still manages to grab the empty json.py file somehow (AttributeError: 'module' object has no attribute 'loads').
My understanding was that the package version of json here should have only been accessible by namespace, i.e. from app import json, but the namespacing doesn't seem to work here.
On python3, I can not reproduce the issue. I also noticed if we put from __future__ import absolute_import into the example.py file, the problem just goes away.
How does the import statement find the local file, and why does it shadow the core library version?
edit: on another minor note, by the time we have reached the line import json, there is already a json module loaded into sys.modules from the lines above. So why does python try to import the module again, shouldn't it simply use the one already in the module cache?

You have more or less arrived at the answer here. Python 2.x by default will do a package-relative import first, which includes the potential of "shadowing" a base-level package.
See the section on Intra-package References in the python 2 documentation.
The ability to specify explicit relative imports, as well as from __future__ import absolute_import, were actually introduced back in Python 2.5 and this is explained further in PEP 328. This behavior became the default in Python 3. The new behavior (assumed absolute and explicit relative imports) was implemented in large part explicitly to address the problem you raise (shadowing builtin modules) although it also allows greater control with multi-level relative import syntax (i.e. .. for the parent module, ... for a level further up, and so on.)

python import package globally

My directory structure is:
package_name/
__init__.py
script.py
time.py
I'm trying to import time (the built-in package) from script.py. however, my sibling time.py is hooked up instead.
What's the syntax of importing a package globally? (e.g. importing from /usr/lib/python2.7/dist-packages)
In c# i know it as the global:: prefix, what's the equivalent in python?

Be more explicit with your imports.
import time will import the python time module.
from mypackage import time will import the time module in your mypackage package.
import time should never import mypackage.time
That said, don't shadow python built-in names, it's a bad habit and leads to these headaches later.

A Hack
While there are other ways, i adopted this one
1) add a sub package
Let's call it utils. your directory tree should look like:
package_name/
__init__.py
script.py
time.py
utils/
__init__.py
2) create a file named builtin.py under that package
This should be the contents of builtin.py :
import time
time = time
3) use it!
Import time in script.py using this:
from utils.builtin import time
time.sleep(5)