I have the following setup of a library in python:
library
| - __init__.py
| - lib1.py
| - ...
| - tools
| - __init__.py
| - testlib1.py
so in other words, the directory library contain python modules and the directory tools, which is a subdirectory of library contains e.g. one file testlib1.pyto test the library lib1.py.
testlib1.py therefore need to import lib1.py from the directory above to do some tests etc., just by calling python testlib1.py from somewhere on the computer, assuming the file is in the search path.
In addition, I only want ONE PYTHONPATH to be specified.
But we all know the following idea for testlib1.py does not work because the relative import does not work:
from .. import lib1
...
do something with lib1
I accept two kind of answers:
An answer which describes how still to be possible to call testlib1.py directly as the executing python script.
An answer that explains a better conceptual setup of of the modules etc, with the premise that everything has to be in the directory project and the tools has to be in a different directory than the actual libraries.
If something is not clear I suggest to ask and I will update the question.
Try adding a __init__.py to the tools directory. The relative import should work.
You can't. If you plan to use relative imports then you can't run the module by itself.
Either drop the relative imports or drop the idea of running testlib1.py directly.
Also I believe a test file should never use relative imports. Test should check whether the library works, and thus the code should be as similar as possible to the code that the users would actually use. And usually users do not add files to the library to use relative imports but use absolute imports.
By the way, I think your file organization is too much "java-like": mixing source code and tests. If you want to do tests then why not have something like:
project/
|
+-- src/
| |
| +--library/
| | |
| | +- lib1.py
| | |
| | #...
| +--library2/ #etc.
|
+-- tests/
|
+--testlibrary/
| |
| +- testlib1.py
#etc
To run the tests just use a tool like nosetests which automatically looks for this kind of folder structure and provide tons of options to change the search/test settings.
Actually, I found a solution, which does
works with the current implementation
does not require any changes in PYTHONPATH
without the need to hardcode the top-lebel directory
In testlib1.py the following code will do (has been tested):
import os
import sys
dirparts = os.path.dirname(os.path.abspath(__file__)).split('/')
sys.path.append('/'.join(dirparts[:-1]))
import mylib1
Not exactly sure this is a very clean or straightforward solution, but it allows to import any module from the directory one level up, which is what I need (as the test code or extra code or whetever is always located one level below the actual module files).
Related
Forgive me for another "relative imports" post but I've read every stack overflow post about them and I'm still confused.
I have a file structure that looks something like this:
|---MyProject
| |---constants.py
| |---MyScripts
| | |---script.py
Inside of script.py I have from .. import constants.py
And of course when I run python3 script.py, I get the error ImportError: attempted relative import with no known parent package.
From what I understand, this doesn't work because when you run a python file directly you can't use relative imports. After hours of searching it seems that the most popular solution is to add "../" to sys.path and then do import constants.py but this feels a little hacky. I've got to believe that there is a better solution than this right? Importing constants has got to be a fairly common task.
I've also seen some people recommend adding __init__.py to make the project into a package but I don't really understand how this solves anything?
Are there other solutions out there that I've either missed or just don't understand?
Your library code should live in a package. For your folder to be a package, it needs to have __init__.py.
Your scripts can live in the top directory in development. Here is how you would structure this layout:
|---MyProject
| |---myproject
| | |---__init__.py
| | |---constants.py
| |---script.py
If you were to productionize this, you could place scripts into a special folder, but then they would rely on the package being installed on your system. The layout could look like this:
|---MyProject
| |---myproject
| | |---__init__.py
| | |---constants.py
| |---scripts
| | |---script.py
In both cases your imports would be normal absolute (non-relative) imports like
from myproject.constants import ...
OR
from myproject import constants
You are correct that attempting a relative import or a path modification in a standalone script is hacky. If you want to use the more flexible second layout, make a setup.py install script in the root folder, and run python setup.py develop.
|---MyProject
| |---__init__.py # File1
| |---constants.py
| |---MyScripts
| | |---__init__.py # File 2
| | |---script.py
And then the content of File1 __init__.py is:
from . import constants
from . import MyScripts
And then the content of File2 __init__.py is:
from . import script
By converting MyProject to a package, you would be able to import constants like:
from MyProject import constants
# Rest of your code
After that, you need to add the root of your MyProject to your python path. If you are using PyCharm, you do not need to do anything. By default the following line is added to your Python Console or Debugging sessions:
import sys
sys.path.extend(['path/to/MyProject'])
If you are not using PyCharm, one is is to add the above script before codes you run or for debugging, add that as a code to run before every session, or you define a setup.py for your MyProject package, install it in your environment and nothing else need to be changed.
The syntax from .. import x is only intended to be used inside packages (thus your error message). If you don't want to make a package, the other way to accomplish what you want is to manipulate the path.
In absolute terms, you can do:
import sys
sys.path.append(r'c:\here\is\MyProject')
import constants
where "MyProject" is the folder you described.
If you want to use relative paths, you need to do a little more work:
import inspect
import os
import sys
parent_dir = os.path.split(
os.path.dirname(inspect.getfile(inspect.currentframe())))[0]
sys.path.append(parent_dir)
import constants
To put it concisely, the error message is telling you exactly what's happening; t's just not explaining why. .. isn't a package. It resolves to MyProject but MyProject isn't a package. To make it a package, you must include an “__init__.py” file directly in the “MyProject” directory. It can even be an empty file if you're feeling lazy.
But given what you've shown us, it doesn't seem like your project is supposed to be a package. Try changing the import in script.py to import constants and running python3 MyScripts/script.py from your project directory.
I've seen all sorts of examples and other similar questions, but I can't seem to find an example that exactly matches my scenario. I feel like a total goon asking this because there are so many similar questions, but I just can't seem to get this working "correctly." Here is my project:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
|------- Scripts/
| |
| |----- __init__.py
| |----- CreateUser.py
| |----- FindUser.py
If I move "CreateUser.py" to the main user_management directory, I can easily use: "import Modules.LDAPManager" to import LDAPManager.py --- this works. What I can't do (which I want to do), is keep CreateUser.py in the Scripts subfolder, and import LDAPManager.py. I was hoping to accomplish this by using "import user_management.Modules.LDAPManager.py". This doesn't work. In short, I can get Python files to easily look deeper in the hierarchy, but I can't get a Python script to reference up one directory and down into another.
Note that I am able to solve my problem using:
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
import Modules.LDAPManager as LDAPManager
I've heard that this is bad practice and discouraged.
The files in Scripts are intended to be executed directly (is the init.py in Scripts even necessary?). I've read that in this case, I should be executing CreateUser.py with the -m flag. I've tried some variations on this and just can't seem to get CreateUser.py to recognize LDAPManager.py.
If I move CreateUser.py to the main user_management directory, I can
easily use: import Modules.LDAPManager to import LDAPManager.py
--- this works.
Please, don't. In this way the LDAPManager module used by CreateUser will not be the same as the one imported via other imports. This can create problems when you have some global state in the module or during pickling/unpickling. Avoid imports that work only because the module happens to be in the same directory.
When you have a package structure you should either:
Use relative imports, i.e if the CreateUser.py is in Scripts/:
from ..Modules import LDAPManager
Note that this was (note the past tense) discouraged by PEP 8 only because old versions of python didn't support them very well, but this problem was solved years ago. The current version of PEP 8 does suggest them as an acceptable alternative to absolute imports. I actually like them inside packages.
Use absolute imports using the whole package name(CreateUser.py in Scripts/):
from user_management.Modules import LDAPManager
In order for the second one to work the package user_management should be installed inside the PYTHONPATH. During development you can configure the IDE so that this happens, without having to manually add calls to sys.path.append anywhere.
Also I find it odd that Scripts/ is a subpackage. Because in a real installation the user_management module would be installed under the site-packages found in the lib/ directory (whichever directory is used to install libraries in your OS), while the scripts should be installed under a bin/ directory (whichever contains executables for your OS).
In fact I believe Script/ shouldn't even be under user_management. It should be at the same level of user_management.
In this way you do not have to use -m, but you simply have to make sure the package can be found (this again is a matter of configuring the IDE, installing the package correctly or using PYTHONPATH=. python Scripts/CreateUser.py to launch the scripts with the correct path).
In summary, the hierarchy I would use is:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
Scripts/ (*not* a package)
|
|----- CreateUser.py
|----- FindUser.py
Then the code of CreateUser.py and FindUser.py should use absolute imports to import the modules:
from user_management.Modules import LDAPManager
During installation you make sure that user_management ends up somewhere in the PYTHONPATH, and the scripts inside the directory for executables so that they are able to find the modules. During development you either rely on IDE configuration, or you launch CreateUser.py adding the Scripts/ parent directory to the PYTHONPATH (I mean the directory that contains both user_management and Scripts):
PYTHONPATH=/the/parent/directory python Scripts/CreateUser.py
Or you can modify the PYTHONPATH globally so that you don't have to specify this each time. On unix OSes (linux, Mac OS X etc.) you can modify one of the shell scripts to define the PYTHONPATH external variable, on Windows you have to change the environmental variables settings.
Addendum I believe, if you are using python2, it's better to make sure to avoid implicit relative imports by putting:
from __future__ import absolute_import
at the top of your modules. In this way import X always means to import the toplevel module X and will never try to import the X.py file that's in the same directory (if that directory isn't in the PYTHONPATH). In this way the only way to do a relative import is to use the explicit syntax (the from . import X), which is better (explicit is better than implicit).
This will make sure you never happen to use the "bogus" implicit relative imports, since these would raise an ImportError clearly signalling that something is wrong. Otherwise you could use a module that's not what you think it is.
From Python 2.5 onwards, you can use
from ..Modules import LDAPManager
The leading period takes you "up" a level in your heirarchy.
See the Python docs on intra-package references for imports.
In the "root" __init__.py you can also do a
import sys
sys.path.insert(1, '.')
which should make both modules importable.
I faced the same issues. To solve this, I used export PYTHONPATH="$PWD". However, in this case, you will need to modify imports in your Scripts dir depending on the below:
Case 1: If you are in the user_management dir, your scripts should use this style from Modules import LDAPManager to import module.
Case 2: If you are out of the user_management 1 level like main, your scripts should use this style from user_management.Modules import LDAPManager to import modules.
My Ansible directory structure looks something like this.
Ansible-Repo
|
+-- playbooks
| |
| +-- run_custom_module1
|
+-- library
| |
| +-- custom_module1
| +-- custom_module2
|
+-- bin
| |
| +-- usefulpythonfunctions.py
I want to be able to import usefulpythonfunctions.py from the bin inside my Ansible Module. I have an import usefulpythonfunctions.py at the top of my module, but I receive the following error when I run the playbook.
\r\nImportError: No module named usefulpythonfunctions\r\n", "msg": "MODULE FAILURE", "parsed": false}
Luckily 2.3 release has introduced this feature.
Now you can place your shared code in module_utils folder where your playbook lives and it will be automatically exported by ansible. Also you might be interested in checking the directory structure documentation
You can access your module in module_utils as you would access a standard util
from ansible.module_utils.my_shared_code import MySharedCodeClient
UPDATE
Now you can configure a path where your module utils live using ANSIBLE_MODULE_UTILS env variable. For more details please check documentation
I've been looking into ways to do this and it doesn't appear this is possible in the current version of Ansible.
Honestly, the pull request mentioned by #slackhacker seems to be the cleanest option, but that looks like it may not be available until Ansible 2.1...
Until then...
Deploy your dependencies using ansible
The library you want has to be copied to each system you run on. So, let's just have ansible take care of it for us.
---
tasks:
- name: Copy mylibrary
# Only requirement is that the dst path exists in the default system python path
copy: src=path-to-library dst=/usr/local/lib/python2.7/dist-packages
- name:
mymodule:
arg1: foo
arg2: bar
Download dependencies in your module code
If your module code is accessible to the server you are running on (via HTTP, SSH, etc), you could code the module to go out and grab the dependencies directy in python instead of doing it as a separate task.
This has the added value of not requiring extra steps from the end user, but will probably run slower due to the extra copies having to be made.
Roll your own package
You could also roll your own package using pip or any other packaging toolchain. This is probably not for the faint of heart.
According to my understanding is this not possible in Ansible, but there is pending pull request that offers some hope.
Try creating a folder called usefulpythonfunctions, create an __init__.py and place the functions you want in there.
Additionally, you should be able to create bin/foo/bar.py and use from foo import bar to call functions or classes out of bar.
I've seen all sorts of examples and other similar questions, but I can't seem to find an example that exactly matches my scenario. I feel like a total goon asking this because there are so many similar questions, but I just can't seem to get this working "correctly." Here is my project:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
|------- Scripts/
| |
| |----- __init__.py
| |----- CreateUser.py
| |----- FindUser.py
If I move "CreateUser.py" to the main user_management directory, I can easily use: "import Modules.LDAPManager" to import LDAPManager.py --- this works. What I can't do (which I want to do), is keep CreateUser.py in the Scripts subfolder, and import LDAPManager.py. I was hoping to accomplish this by using "import user_management.Modules.LDAPManager.py". This doesn't work. In short, I can get Python files to easily look deeper in the hierarchy, but I can't get a Python script to reference up one directory and down into another.
Note that I am able to solve my problem using:
sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
import Modules.LDAPManager as LDAPManager
I've heard that this is bad practice and discouraged.
The files in Scripts are intended to be executed directly (is the init.py in Scripts even necessary?). I've read that in this case, I should be executing CreateUser.py with the -m flag. I've tried some variations on this and just can't seem to get CreateUser.py to recognize LDAPManager.py.
If I move CreateUser.py to the main user_management directory, I can
easily use: import Modules.LDAPManager to import LDAPManager.py
--- this works.
Please, don't. In this way the LDAPManager module used by CreateUser will not be the same as the one imported via other imports. This can create problems when you have some global state in the module or during pickling/unpickling. Avoid imports that work only because the module happens to be in the same directory.
When you have a package structure you should either:
Use relative imports, i.e if the CreateUser.py is in Scripts/:
from ..Modules import LDAPManager
Note that this was (note the past tense) discouraged by PEP 8 only because old versions of python didn't support them very well, but this problem was solved years ago. The current version of PEP 8 does suggest them as an acceptable alternative to absolute imports. I actually like them inside packages.
Use absolute imports using the whole package name(CreateUser.py in Scripts/):
from user_management.Modules import LDAPManager
In order for the second one to work the package user_management should be installed inside the PYTHONPATH. During development you can configure the IDE so that this happens, without having to manually add calls to sys.path.append anywhere.
Also I find it odd that Scripts/ is a subpackage. Because in a real installation the user_management module would be installed under the site-packages found in the lib/ directory (whichever directory is used to install libraries in your OS), while the scripts should be installed under a bin/ directory (whichever contains executables for your OS).
In fact I believe Script/ shouldn't even be under user_management. It should be at the same level of user_management.
In this way you do not have to use -m, but you simply have to make sure the package can be found (this again is a matter of configuring the IDE, installing the package correctly or using PYTHONPATH=. python Scripts/CreateUser.py to launch the scripts with the correct path).
In summary, the hierarchy I would use is:
user_management (package)
|
|------- __init__.py
|
|------- Modules/
| |
| |----- __init__.py
| |----- LDAPManager.py
| |----- PasswordManager.py
|
Scripts/ (*not* a package)
|
|----- CreateUser.py
|----- FindUser.py
Then the code of CreateUser.py and FindUser.py should use absolute imports to import the modules:
from user_management.Modules import LDAPManager
During installation you make sure that user_management ends up somewhere in the PYTHONPATH, and the scripts inside the directory for executables so that they are able to find the modules. During development you either rely on IDE configuration, or you launch CreateUser.py adding the Scripts/ parent directory to the PYTHONPATH (I mean the directory that contains both user_management and Scripts):
PYTHONPATH=/the/parent/directory python Scripts/CreateUser.py
Or you can modify the PYTHONPATH globally so that you don't have to specify this each time. On unix OSes (linux, Mac OS X etc.) you can modify one of the shell scripts to define the PYTHONPATH external variable, on Windows you have to change the environmental variables settings.
Addendum I believe, if you are using python2, it's better to make sure to avoid implicit relative imports by putting:
from __future__ import absolute_import
at the top of your modules. In this way import X always means to import the toplevel module X and will never try to import the X.py file that's in the same directory (if that directory isn't in the PYTHONPATH). In this way the only way to do a relative import is to use the explicit syntax (the from . import X), which is better (explicit is better than implicit).
This will make sure you never happen to use the "bogus" implicit relative imports, since these would raise an ImportError clearly signalling that something is wrong. Otherwise you could use a module that's not what you think it is.
From Python 2.5 onwards, you can use
from ..Modules import LDAPManager
The leading period takes you "up" a level in your heirarchy.
See the Python docs on intra-package references for imports.
In the "root" __init__.py you can also do a
import sys
sys.path.insert(1, '.')
which should make both modules importable.
I faced the same issues. To solve this, I used export PYTHONPATH="$PWD". However, in this case, you will need to modify imports in your Scripts dir depending on the below:
Case 1: If you are in the user_management dir, your scripts should use this style from Modules import LDAPManager to import module.
Case 2: If you are out of the user_management 1 level like main, your scripts should use this style from user_management.Modules import LDAPManager to import modules.
I have a directory struture like that:
project
| __init__.py
| project.py
| src/
| __init__.py
| class_one.py
| class_two.py
| test/
| __init__.py
| test_class_one.py
Which project.py just instantiate ClassOne and run it.
My problem is in the tests, I don't know how to import src classes. I've tried importing these ways and I got nothing:
from project.src.class_one import ClassOne
and
from ..src.class_one import ClassOne
What am I doing wrong? Is there a better directory structure?
----- EDIT -----
I changed my dir structure, it's like this now:
Project/
| project.py
| project/
| __init__.py
| class_one.py
| class_two.py
| test/
| __init__.py
| test_class_one.py
And in the test_class_one.py file I'm trying to import this way:
from project.class_one import ClassOne
And it still doesn't work. I'm not using the executable project.py inside a bin dir exactly because I can't import a package from a higher level dir. :(
Thanks. =D
It all depends on your python path. The easiest way to achieve what you're wanting to do here is to set the PYTHONPATH environment variable to where the "project" directory resides. For example, if your source is living in:
/Users/crocidb/src/project/
I would:
export PYTHONPATH=/Users/crocidb/src
and then in the test_one.py I could:
import project.src.class_one
Actually I would probably do it this way:
export PYTHONPATH=/Users/crocidb/src/project
and then this in test_one.py:
import src.class_one
but that's just my preference and really depends on what the rest of your hierarchy is. Also note that if you already have something in PYTHONPATH you'll want to add to it:
export PYTHONPATH=/Users/crocidb/src/project:$PYTHONPATH
or in the other order if you want your project path to be searched last.
This all applies to windows, too, except you would need to use windows' syntax to set the environment variables.
From Jp Calderone's excellent blog post:
Do:
name the directory something related to your project. For example, if your
project is named "Twisted", name the
top-level directory for its source
files Twisted. When you do releases,
you should include a version number
suffix: Twisted-2.5.
create a directory Twisted/bin and put your executables there, if you
have any. Don't give them a .py
extension, even if they are Python
source files. Don't put any code in
them except an import of and call to a
main function defined somewhere else
in your projects. (Slight wrinkle:
since on Windows, the interpreter is
selected by the file extension, your
Windows users actually do want the .py
extension. So, when you package for
Windows, you may want to add it.
Unfortunately there's no easy
distutils trick that I know of to
automate this process. Considering
that on POSIX the .py extension is a
only a wart, whereas on Windows the
lack is an actual bug, if your
userbase includes Windows users, you
may want to opt to just have the .py
extension everywhere.)
If your project is expressable as a single Python source file, then put it
into the directory and name it
something related to your project. For
example, Twisted/twisted.py. If you
need multiple source files, create a
package instead (Twisted/twisted/,
with an empty
Twisted/twisted/__init__.py) and place
your source files in it. For example,
Twisted/twisted/internet.py.
put your unit tests in a sub-package of your package (note - this means
that the single Python source file
option above was a trick - you always
need at least one other file for your
unit tests). For example,
Twisted/twisted/test/. Of course, make
it a package with
Twisted/twisted/test/__init__.py.
Place tests in files like
Twisted/twisted/test/test_internet.py.
add Twisted/README and Twisted/setup.py to explain and
install your software, respectively,
if you're feeling nice.
Don't:
put your source in a directory called src or lib. This makes it hard
to run without installing.
put your tests outside of your Python package. This makes it hard to
run the tests against an installed
version.
create a package that only has a __init__.py and then put all your code into __init__.py. Just make a module
instead of a package, it's simpler.
try to come up with magical hacks to make Python able to import your module
or package without having the user add
the directory containing it to their
import path (either via PYTHONPATH or
some other mechanism). You will not
correctly handle all cases and users
will get angry at you when your
software doesn't work in their
environment.