I'm trying to build a simple project in Python,however I can't seem to wrap my head around how it's done despite reading the documentations over and over and reading example code.
I'm semi-new to Python, having a background of only single-file scripting.
However I made numerous Node.js and Java projects with multiple folders and packages.
In Python However I can't seem to grasp the project structure and workflow mechanisms- init.py, virtualenvs, setuptools etc. it all seems very alien and unintuitive to me, hurting my overall productivity(ironically Python is supposed to be a language aimed at productivity, right?)
My project has this example structure:
package_test (package)
|
|------- __init__.py
|------- main.py (entry point)
|
|------- /lib (modules)
| |
| |----- __init__.py
| |----- lib.py
|
|------- /Tests
| |
| |----- __init__.py
| |----- test.py
in main.py:
from lib import lib
lib.libtest()
in lib.py:
def libtest():
print("libtest")
in test.py:
from lib import lib
lib.libtest()
Running main.py works, However when running test.py it cannot find the module. I have tried solutions such as appending to sys.path, putting '..' before lib, and more- none have worked.
This is just an example, but I wish to develop more complex projects in Python with multiple subfolders in the future(I think Python has some cool features and nice libraries), yet this problem keeps bothering me. I never had to think about these issues when developing in Java or in Node, not to mention stuff such as virtualenv etc.
Thank you in Advance
Your test.py needs to know lib.py is one level above it and inside the lib folder. So all you should need to do is append the path of your lib folder before running the import of lib.py into test.py. Something like
import sys
sys.path.append("..//lib")
import lib
Please note that the answer above assumes you run on Windows. A more comprehensive approach would be to replace the windows paths with proper os.path syntax. Also another assumption made by this code is that you will run your test.py from it's folder, not with an absolute/relative path.
import sys
import os.path
sys.path.append(os.path.join(os.path.pardir, 'lib'))
import lib
If you want to run your test.py with a path, your code could look like this:
import sys
import os.path
# os.path.abspath(__file__) gets the absolute path of the current python scrip
# os.path.dirname gets the absolute path of whatever file you ask it to (minus the filename)
# The line below appends the full path of your lib folder to the current script's path
sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), os.path.pardir, 'lib'))
# This should allow test.py to import libtest regardless of where your run it from. Assuming your provided project structure.
from lib import libtest
libtest()
Related
Forgive me for another "relative imports" post but I've read every stack overflow post about them and I'm still confused.
I have a file structure that looks something like this:
|---MyProject
| |---constants.py
| |---MyScripts
| | |---script.py
Inside of script.py I have from .. import constants.py
And of course when I run python3 script.py, I get the error ImportError: attempted relative import with no known parent package.
From what I understand, this doesn't work because when you run a python file directly you can't use relative imports. After hours of searching it seems that the most popular solution is to add "../" to sys.path and then do import constants.py but this feels a little hacky. I've got to believe that there is a better solution than this right? Importing constants has got to be a fairly common task.
I've also seen some people recommend adding __init__.py to make the project into a package but I don't really understand how this solves anything?
Are there other solutions out there that I've either missed or just don't understand?
Your library code should live in a package. For your folder to be a package, it needs to have __init__.py.
Your scripts can live in the top directory in development. Here is how you would structure this layout:
|---MyProject
| |---myproject
| | |---__init__.py
| | |---constants.py
| |---script.py
If you were to productionize this, you could place scripts into a special folder, but then they would rely on the package being installed on your system. The layout could look like this:
|---MyProject
| |---myproject
| | |---__init__.py
| | |---constants.py
| |---scripts
| | |---script.py
In both cases your imports would be normal absolute (non-relative) imports like
from myproject.constants import ...
OR
from myproject import constants
You are correct that attempting a relative import or a path modification in a standalone script is hacky. If you want to use the more flexible second layout, make a setup.py install script in the root folder, and run python setup.py develop.
|---MyProject
| |---__init__.py # File1
| |---constants.py
| |---MyScripts
| | |---__init__.py # File 2
| | |---script.py
And then the content of File1 __init__.py is:
from . import constants
from . import MyScripts
And then the content of File2 __init__.py is:
from . import script
By converting MyProject to a package, you would be able to import constants like:
from MyProject import constants
# Rest of your code
After that, you need to add the root of your MyProject to your python path. If you are using PyCharm, you do not need to do anything. By default the following line is added to your Python Console or Debugging sessions:
import sys
sys.path.extend(['path/to/MyProject'])
If you are not using PyCharm, one is is to add the above script before codes you run or for debugging, add that as a code to run before every session, or you define a setup.py for your MyProject package, install it in your environment and nothing else need to be changed.
The syntax from .. import x is only intended to be used inside packages (thus your error message). If you don't want to make a package, the other way to accomplish what you want is to manipulate the path.
In absolute terms, you can do:
import sys
sys.path.append(r'c:\here\is\MyProject')
import constants
where "MyProject" is the folder you described.
If you want to use relative paths, you need to do a little more work:
import inspect
import os
import sys
parent_dir = os.path.split(
os.path.dirname(inspect.getfile(inspect.currentframe())))[0]
sys.path.append(parent_dir)
import constants
To put it concisely, the error message is telling you exactly what's happening; t's just not explaining why. .. isn't a package. It resolves to MyProject but MyProject isn't a package. To make it a package, you must include an “__init__.py” file directly in the “MyProject” directory. It can even be an empty file if you're feeling lazy.
But given what you've shown us, it doesn't seem like your project is supposed to be a package. Try changing the import in script.py to import constants and running python3 MyScripts/script.py from your project directory.
I know this is a worn out topic but the import mechanism/s in python is still confusing the masses. What I want is the ability to import a custom module that is in a parent directory in a way that allows me to take a project to another environment and have all of the imports work.
For example a structure like this:
repo
|--- folder1
| |--- script1.py
|--- folder2
| |--- script2.py
|--- utils
|--- some-util.py
How can I import from some-util.py in both script1 and script2? The idea is that I could clone the repo into a remote host and run scripts from folders 1 and 2 that may have the shared dependency of some-util.py only I don't want to have to run anything before hand. I want to be able to:
connect to box
git clone repo
python repo/folder1/script1.py
contents of script1 and script2:
import some-util
<code>
EDIT:
I forgot to mention that occasionally the scripts need to be run from another directory like:
/nas/some_folder/repo/folder1/script1.py args..
Also, the box is limited to python 2.7.5
The trick is to implement as your scripts as modules (read here and here for an overview of what the python -m switch means).
Here is a structure, also notice every directory contains an (empty file) named __init__.py:
repo/
|____utils/
| |____someutil.py
| |___ __init__.py
|___ __init__.py
|____folder1/
|____script1.py
|___ __init__.py
utils.someutil may contain something like this:
def say_hello():
return "Hello World."
And your script1.py may contain something like:
from ..utils.someutil import say_hello
if __name__ == "__main__":
print(say_hello())
Then running the following:
python -m repo.folder1.script1
... produces:
Hello World.
I've seen all sorts of examples and other similar questions, but I can't seem to find an example that exactly matches my scenario. I feel like a total goon asking this because there are so many similar questions, but I just can't seem to get this working "correctly." Here is my project:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
|------- Scripts/
| |
| |----- __init__.py
| |----- CreateUser.py
| |----- FindUser.py
If I move "CreateUser.py" to the main user_management directory, I can easily use: "import Modules.LDAPManager" to import LDAPManager.py --- this works. What I can't do (which I want to do), is keep CreateUser.py in the Scripts subfolder, and import LDAPManager.py. I was hoping to accomplish this by using "import user_management.Modules.LDAPManager.py". This doesn't work. In short, I can get Python files to easily look deeper in the hierarchy, but I can't get a Python script to reference up one directory and down into another.
Note that I am able to solve my problem using:
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
import Modules.LDAPManager as LDAPManager
I've heard that this is bad practice and discouraged.
The files in Scripts are intended to be executed directly (is the init.py in Scripts even necessary?). I've read that in this case, I should be executing CreateUser.py with the -m flag. I've tried some variations on this and just can't seem to get CreateUser.py to recognize LDAPManager.py.
If I move CreateUser.py to the main user_management directory, I can
easily use: import Modules.LDAPManager to import LDAPManager.py
--- this works.
Please, don't. In this way the LDAPManager module used by CreateUser will not be the same as the one imported via other imports. This can create problems when you have some global state in the module or during pickling/unpickling. Avoid imports that work only because the module happens to be in the same directory.
When you have a package structure you should either:
Use relative imports, i.e if the CreateUser.py is in Scripts/:
from ..Modules import LDAPManager
Note that this was (note the past tense) discouraged by PEP 8 only because old versions of python didn't support them very well, but this problem was solved years ago. The current version of PEP 8 does suggest them as an acceptable alternative to absolute imports. I actually like them inside packages.
Use absolute imports using the whole package name(CreateUser.py in Scripts/):
from user_management.Modules import LDAPManager
In order for the second one to work the package user_management should be installed inside the PYTHONPATH. During development you can configure the IDE so that this happens, without having to manually add calls to sys.path.append anywhere.
Also I find it odd that Scripts/ is a subpackage. Because in a real installation the user_management module would be installed under the site-packages found in the lib/ directory (whichever directory is used to install libraries in your OS), while the scripts should be installed under a bin/ directory (whichever contains executables for your OS).
In fact I believe Script/ shouldn't even be under user_management. It should be at the same level of user_management.
In this way you do not have to use -m, but you simply have to make sure the package can be found (this again is a matter of configuring the IDE, installing the package correctly or using PYTHONPATH=. python Scripts/CreateUser.py to launch the scripts with the correct path).
In summary, the hierarchy I would use is:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
Scripts/ (*not* a package)
|
|----- CreateUser.py
|----- FindUser.py
Then the code of CreateUser.py and FindUser.py should use absolute imports to import the modules:
from user_management.Modules import LDAPManager
During installation you make sure that user_management ends up somewhere in the PYTHONPATH, and the scripts inside the directory for executables so that they are able to find the modules. During development you either rely on IDE configuration, or you launch CreateUser.py adding the Scripts/ parent directory to the PYTHONPATH (I mean the directory that contains both user_management and Scripts):
PYTHONPATH=/the/parent/directory python Scripts/CreateUser.py
Or you can modify the PYTHONPATH globally so that you don't have to specify this each time. On unix OSes (linux, Mac OS X etc.) you can modify one of the shell scripts to define the PYTHONPATH external variable, on Windows you have to change the environmental variables settings.
Addendum I believe, if you are using python2, it's better to make sure to avoid implicit relative imports by putting:
from __future__ import absolute_import
at the top of your modules. In this way import X always means to import the toplevel module X and will never try to import the X.py file that's in the same directory (if that directory isn't in the PYTHONPATH). In this way the only way to do a relative import is to use the explicit syntax (the from . import X), which is better (explicit is better than implicit).
This will make sure you never happen to use the "bogus" implicit relative imports, since these would raise an ImportError clearly signalling that something is wrong. Otherwise you could use a module that's not what you think it is.
From Python 2.5 onwards, you can use
from ..Modules import LDAPManager
The leading period takes you "up" a level in your heirarchy.
See the Python docs on intra-package references for imports.
In the "root" __init__.py you can also do a
import sys
sys.path.insert(1, '.')
which should make both modules importable.
I faced the same issues. To solve this, I used export PYTHONPATH="$PWD". However, in this case, you will need to modify imports in your Scripts dir depending on the below:
Case 1: If you are in the user_management dir, your scripts should use this style from Modules import LDAPManager to import module.
Case 2: If you are out of the user_management 1 level like main, your scripts should use this style from user_management.Modules import LDAPManager to import modules.
I've seen all sorts of examples and other similar questions, but I can't seem to find an example that exactly matches my scenario. I feel like a total goon asking this because there are so many similar questions, but I just can't seem to get this working "correctly." Here is my project:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
|------- Scripts/
| |
| |----- __init__.py
| |----- CreateUser.py
| |----- FindUser.py
If I move "CreateUser.py" to the main user_management directory, I can easily use: "import Modules.LDAPManager" to import LDAPManager.py --- this works. What I can't do (which I want to do), is keep CreateUser.py in the Scripts subfolder, and import LDAPManager.py. I was hoping to accomplish this by using "import user_management.Modules.LDAPManager.py". This doesn't work. In short, I can get Python files to easily look deeper in the hierarchy, but I can't get a Python script to reference up one directory and down into another.
Note that I am able to solve my problem using:
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
import Modules.LDAPManager as LDAPManager
I've heard that this is bad practice and discouraged.
The files in Scripts are intended to be executed directly (is the init.py in Scripts even necessary?). I've read that in this case, I should be executing CreateUser.py with the -m flag. I've tried some variations on this and just can't seem to get CreateUser.py to recognize LDAPManager.py.
If I move CreateUser.py to the main user_management directory, I can
easily use: import Modules.LDAPManager to import LDAPManager.py
--- this works.
Please, don't. In this way the LDAPManager module used by CreateUser will not be the same as the one imported via other imports. This can create problems when you have some global state in the module or during pickling/unpickling. Avoid imports that work only because the module happens to be in the same directory.
When you have a package structure you should either:
Use relative imports, i.e if the CreateUser.py is in Scripts/:
from ..Modules import LDAPManager
Note that this was (note the past tense) discouraged by PEP 8 only because old versions of python didn't support them very well, but this problem was solved years ago. The current version of PEP 8 does suggest them as an acceptable alternative to absolute imports. I actually like them inside packages.
Use absolute imports using the whole package name(CreateUser.py in Scripts/):
from user_management.Modules import LDAPManager
In order for the second one to work the package user_management should be installed inside the PYTHONPATH. During development you can configure the IDE so that this happens, without having to manually add calls to sys.path.append anywhere.
Also I find it odd that Scripts/ is a subpackage. Because in a real installation the user_management module would be installed under the site-packages found in the lib/ directory (whichever directory is used to install libraries in your OS), while the scripts should be installed under a bin/ directory (whichever contains executables for your OS).
In fact I believe Script/ shouldn't even be under user_management. It should be at the same level of user_management.
In this way you do not have to use -m, but you simply have to make sure the package can be found (this again is a matter of configuring the IDE, installing the package correctly or using PYTHONPATH=. python Scripts/CreateUser.py to launch the scripts with the correct path).
In summary, the hierarchy I would use is:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
Scripts/ (*not* a package)
|
|----- CreateUser.py
|----- FindUser.py
Then the code of CreateUser.py and FindUser.py should use absolute imports to import the modules:
from user_management.Modules import LDAPManager
During installation you make sure that user_management ends up somewhere in the PYTHONPATH, and the scripts inside the directory for executables so that they are able to find the modules. During development you either rely on IDE configuration, or you launch CreateUser.py adding the Scripts/ parent directory to the PYTHONPATH (I mean the directory that contains both user_management and Scripts):
PYTHONPATH=/the/parent/directory python Scripts/CreateUser.py
Or you can modify the PYTHONPATH globally so that you don't have to specify this each time. On unix OSes (linux, Mac OS X etc.) you can modify one of the shell scripts to define the PYTHONPATH external variable, on Windows you have to change the environmental variables settings.
Addendum I believe, if you are using python2, it's better to make sure to avoid implicit relative imports by putting:
from __future__ import absolute_import
at the top of your modules. In this way import X always means to import the toplevel module X and will never try to import the X.py file that's in the same directory (if that directory isn't in the PYTHONPATH). In this way the only way to do a relative import is to use the explicit syntax (the from . import X), which is better (explicit is better than implicit).
This will make sure you never happen to use the "bogus" implicit relative imports, since these would raise an ImportError clearly signalling that something is wrong. Otherwise you could use a module that's not what you think it is.
From Python 2.5 onwards, you can use
from ..Modules import LDAPManager
The leading period takes you "up" a level in your heirarchy.
See the Python docs on intra-package references for imports.
In the "root" __init__.py you can also do a
import sys
sys.path.insert(1, '.')
which should make both modules importable.
I faced the same issues. To solve this, I used export PYTHONPATH="$PWD". However, in this case, you will need to modify imports in your Scripts dir depending on the below:
Case 1: If you are in the user_management dir, your scripts should use this style from Modules import LDAPManager to import module.
Case 2: If you are out of the user_management 1 level like main, your scripts should use this style from user_management.Modules import LDAPManager to import modules.
stuff/
__init__.py
mylib.py
Foo/
__init__.py
main.py
foo/
__init__.py
script.py
script.py wants to import mylib.py
This is just an example, but really I just want to do a relative import of a module in a parent directory. I've tried various things and get this error...
Attempted relative import beyond toplevel package
I read somewhere that the script from where the program starts shouldn't in the package, and I tried modifying the structure for that like so...
stuff/
mylib.py
foo.py // equivalent of main.py in above
foo/
__init__.py
script.py
but got same error.
How can I accomplish this? Is this even an adequate approach?
Edit: In Python 2
After fiddling with it a bit more, I realized how to set it up, and for the sake of specificity I won't use foo bar names. My project directory is set up as...
tools/
core/
object_editor/
# files that need to use ntlib.py
editor.py # see example at bottom
__init__.py
state_editor/
# files that need to use ntlib.py
__init__.py
ntlib.py
__init__.py # core is the top level package
LICENSE
state_editor.py # equivalent to main.py for the state editor
object_editor.py # equivalent to main.py for the object editor
A line in object_editor.py looks like...
from core.object_editor import editor
A line in editor.py looks like...
from .. import ntlib
or alternatively
from core import ntlib
The key is that in the example I gave in the question, the "main" script was being run from within the package. Once I moved it out, created a specific package (core), and moved the library I wanted the editors to share (ntlib) into that package, everything was hunky-dory.
though as long "stuff" is not in your python PATH you got no choice than adding the path.
If you know the level of your script.py from stuff you can do for example:
import sys
import os
sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..'))
I'm running Python 3.4.2 on Windows 7 and tore my hair out over this.
When running either of these:
python -m unittest
python -m unittest discover
...I would get the 'Attempted relative import beyond toplevel package' error.
For me, the solution was dropping the ".." in my [test_stock.py].
The line was:
from ..stock import Stock
Changed it to:
from stock import Stock
.. and it works.
Folder structure:
C:\
|
+-- stock_alerter
|
+-- __init__.py
+-- stock.py
|
\-- tests
|
+-- __init__.py
\-- test_stock.py
From the PEP it appears that you cannot use a relative import to import a file that is not packaged.
So you would need to add a __init__.py to stuff and change your imports to something like from .mylib import *
However, the PEP seems to make no allowance to keep mylib packaged up in a module. So you might be required to change how you call your library functions.
Another alternative is to move mylib into a subpackage and import it as from .libpackage import mylib
If you're on Linux or perhaps a similar *nix, you can hack this with symlinks.
stuff/
mylib.py
foo.py // equivalent of main.py in above
foo/
script.py
mylib.py -> ../mylib.py
foo2/
script2.py
mylib.py -> ../mylib.py
This is likely not a good pattern to follow.
In my case I opted for it because I had multiple executables dependent on the same library that needed to be put into separate directories.
Implementation of new executable tests shouldn't require the test writer to have a deep understanding of python imports.
tests/
common/
commonlib.py
test1/
executable1.py
executable2.py
commonlib.py -> ../common/commonlib.py
test2/
executable1.py
executable2.py
commonlib.py -> ../common/commonlib.py