I have a python app, that I'm developing. There is a need to use another library, that resides in different directory.
The file layout looks like this:
dir X has two project dirs:
current-project
xLibrary
I'd like to use xLibrary in currentProject. I've been trying writting code as if all the sources resided in the same directory and calling my projects main script with:
PYTHONPATH=.:../xLibrary ./current-project.py
but this does not work. I'd like to use its code base without installing the library globaly or copying it to my project's directory. Is it possible? Or if not, how should I deal with this problem.
It's generally a good programming practice to isolate packages into actual packages and treat them as such. If you're sure you'd like to continue with that approach though you can modify the search path from within python via:
import sys
sys.path.append( "<path_containing_the_other_python_files>" )
To avoid embedding absolute paths, you can use os.path.abspath(__file__) to obtain the absolute path to the currently executing .py file and follow up with a few os.path.dirname() calls to construct the proper relative path for inclusion to sys.path
A slightly altered approach that would allow you to get the best of both worlds would be to add an __init__.py file to xLibrary's directory then add the path containing 'xLibrary' to sys.path instead. Subsequent Python code could then import everything "properly" via from xLibrary import my_module rather than just import my_module which could be confusing to people accustomed to the standard package directory layout.
This depends how you use xLibrary from current-project.
If you do something like from xLibrary import module1 inside current-project, the xLibrary needs to be laid out as a Python package:
xLibrary/
xLibrary/__init__.py
xLibrary/module1.py # or whatever other modules the package consists of
In this case, you should include xLibrary's parent directory in PYTHONPATH:
PYTHONPATH=.:.. ./current-project.py
However, if xLibrary is just a collection of Python modules that you import individually (that is, if you do import module1 and import module2 ìn current-project, with xLibrary containing the files module1.py and module2.py), you should include xLibrary in PYTHONPATH just as you did:
PYTHONPATH=.:../xLibrary ./current-project.py
bash$ ln -s ../xLibrary xLibrary
First, it is probably better to use absolute paths in your PYTHONPATH variable.
Second, I don't think you need to add current directory to the path.
Other than that, it would be good to know what it is that doesn't work and what the error message is.
The command line you have there seems to be missing a semicolon
Try these two:
a=12 echo $a
b=12 ;echo $b
...and you'll see the difference.
Apart from the other suggestions, you may consider the virtualenv package. After writing a little setup.py file you can "virtually install" the library and avoid all the PYTHONPATH munging in general.
This is a really good practice and is encouraged by the python community.
Otherwise I prefer the use of sys.path method (by Rakis)
Related
I know that python mechanism doesn't allow doing relative imports without a known parent package, but I want to know which the reason is.
If relative imports worked without known parent package it would make developers life much easier (I think, correct me if I am wrong)
Example:
From a python script a have to import a global definitions file. Right now I have to do it like this:
DEFS_PATH = '../../../'
sys.path.append(DEFS_PATH)
import DEFINITIONS as defs
If I could import this file just like this without having to specify the -m flag when executing the script or creating a __init__.py file that collects all packages. It would make everything much more easier.
from .... import DEFINITIONS as defs
Of course doing this raises the famous import error:
Obviously this is a toy example, but imagine having to repeat this in hundreds of python scripts...
Is there any workaround for importing relative packages without a known parent package that doesn't involve tha hacky ugly way (sys.path.append(...) or python -m myscript )?
I solved this problem but in a different way. I have a folder where I have lots of global functions that I used in different packages, I could create a Python Package of this folder, however I would have to rebuild it each time I changed something.
The solution that fitted me was to add user-packages.pth file in site-packages directory of my current environment, but it could also be added to global site-packages folder. Inside this user-packages.pth I added the absolute path to my directory where all the global utils are. And now I just have to do from any python script
from utils import data_processing as dp
from utils.database import database_connection as dc
Now I don't need to add in each file sys.path.append("path/to/myutils/")
Note:
The .pth file could have any file name (customName.pth) and paths inside the file should be separated by carriage return ("\n"). Also, paths should be absoulte.
For example:
C:\path\to\utils1
C:\path\to\other\utils2
So Im a beginner to python/programming and came upon this code in a tutorial, which Im having trouble understanding.
from pythonds.basic.stack import Stack
What I did was , I went to the site-packages folder in my python directory (which holds all modules). There I could find the directory structure to be : -
pythonds/basic/stack.py
The file stack.py has a "class Stack" inside it.
So am I correct in interpreting/relating the import command to this directory structure ?
Also , whenever such a long chaining of modules happen in python, can it always be understood in such a manner.
In command line, you can do like this:
C:\Python27\Lib>pip intall pythonds
Then this module can work.
Not all the time.
It's probably better to not try and compare the directory structure with the module path, unless you have to debug modules or install them manually.
Sometimes, your PYTHONPATH will be extended to include subdirectories in site-packages, and then there'll be an extra subdirectory.
Other times, there can be an __init__.py file in the pythonds/basic/ directory (there likely is), that can contain
from .stack import Stack
in which case the import path could be
from pythonds.basic import Stack
Your understanding is right.
import pythonds.basic.stack
This will make all the classes in the module accessible by your script. Whereas,
from pythonds.basic.stack import Stack
will make only the Stack class accessible by your script.
I have the following structure on a Python program:
my_program/
main.py
packages/
__init.py__
package_to_share/
__init__.py
main_to_share.py
module_to_share.py
package_A/
__init__.py
main_A.py
some_module_A.py
package_B/
__init__.py
main_B.py
some_module_B.py
The package package_to_share provides functionality that every package in the packages folder uses and that main.py at the root folder uses.
I also want to be able to cd into each package and be able to run main_X.py.
So far I figured out how to access functionality from main.py:
import packages.package_A.some_module_A
import packages.package_to_share.module_to_share
but I am having problems accessing the functionality in package_to_share from regular packages (e.g. package_A)
For example, when in main_A.py or some_module_A.py, typing import packages.package_to_share.module_to_share fails.
This leads me to following questions questions:
Given the specifics of my problem, with packages to be shared (accessed) by files at the root folder and by other packages, is there a better way to organize my folders ? Does this organization of modules and files in general conform to good standards in Python?
The following looks incredibly hacky to me, but it's the best thing I came up with to make sure my regular modules see the "shared" modules:
p_this_file = os.path.dirname(os.path.realpath(__file__))
new_sys_path.append(os.path.join(p_cwd, '..')
new_sys_path.extend(sys.path)
sys.path = new_sys_path
It also does not prevent my regular packages from importing each other.
Rather than manipulating the path for imports (which I don't recommend), you could use relative imports within module_A:
from .. import shared
That is what I sometimes do, though I generally have my packages installed so I can reference them fully where I need to use them:
from my_module import shared
I have a package in my PYTHONPATH that looks something like this:
package/
__init__.py
module.py
print 'Loading module'
If I'm running Python from the package/ directory (or writing another module in this directory) and type
import module
it loads module.py and prints out "Loading module" as expected. However, if I then type
from package import module
it loads module.py and prints "Loading module" again, which I don't expect. What's the rationale for this?
Note: I think I understand technically why Python is doing this, because the sys.modules key for import module is just "module", but for from package import module it's "package.module". So I guess what I want to know is why the key is different here -- why isn't the file's path name used as the key so that Python does what one expects here?
Effectively, by running code from the package directory, you've misconfigured Python. You shouldn't have put that directory on sys.path, since it's inside a package.
Python doesn't use the filename as the key because it's not importing a file, it's importing a module. Allowing people to do 'import c:\jim\my files\projects\code\stuff' would encourage all kinds of nastiness.
Consider this case instead: what if you were in ~/foo/package/ and ~/bar were on PYTHONPATH - but ~/bar is just a symlink to ~/foo? Do you expect Python to resolve, then deduplicate the symbolic link for you? What if you put a relative directory on PYTHONPATH, then change directories? What if 'foo.py' is a symlink to 'bar.py'? Do you expect both of those to be de-duplicated too? What if they're not symlinks, but just exact copies? Adding complex rules to try to do something convenient in ambiguous circumstances means it does something highly inconvenient for other people. (Python zen 12: in the face of ambiguity, refuse the temptation to guess.)
Python does something simple here, and it's your responsibility to make sure that the environment is set up correctly. Now, you could argue that it's not a very good idea to put the current directory on PYTHONPATH by default - I might even agree with you - but given that it is there, it should follow the same consistent set of rules that other path entries do. If it's intended to be run from an arbitrary directory, your application can always remove the current directory from sys.path by starting off with sys.path.remove('').
It is a minor defect of the current module system.
When importing module, you do it from the current namespace, which has no name. the values inside this namespace are the same as those in package, but the interpreter cannot know it.
When importing package.module, you import module from the package namespace.
This the reason, that the main.py should be outside the package forlder.
Many modules have this organisation :
package /
main.py
package /
sub_package1/
sub_package2/
sub_package3/
module1.py
module2.py
Calling only main.py make sure the namespaces are correctly set, aka the current namespace is main.py's. Its makes impossible to call import module1.py in module2.py. You'ld need to call import package.module1. Makes things simpler and homogeneous.
And yes, import the current folder as the current nameless folder was a bad idea.
It is a PITA if you go beyond a few scripts. But as Python started there, it was not completely senseless.
I'm using Python 2.7. I'm rather new to the python langauge. I have two python modules - "Trailcrest.py" and "Glyph.py", both in the same folder, but not in the Python27 folder.
I need to import "Trailcrest.py" into "Glyph.py", but I am getting the message that "no such module exists".
Additionally, whatever means I use to import the module needs to not be dependent on a solid-state path. This program is cross-platform, and the path can be changed depending on the user's preferences. However, these two modules will always be in the same folder together.
How do I do this?
If you have Trailcrest.py and Glyph.py in the same folder, importing one into the other is as simple as:
import Trailcrest
import Glyph
If this does not work, there seems to be something wrong with your Python setup. You might want to check what's in sys.path.
import sys
print sys.path
To elaborate a bit on Ferdinand Beyer's answer, sys.path is a list of file locations that the default module importer checks. Some, though not all installations of python will add the current directory or the directory of the __main__ module to the path. To make sure that the paths relative to a given module are importable in that module, do something like this:
import os.path, sys
sys.path.append(os.path.dirname(__file__))
But something like that shouldn't ever make it into a "production" product. Instead, use something like distutils to install the module's package into the python site-packages directory.
This can also be achieved using the environment variable PYTHONPATH which also influences Python's search path. This can be done in a shell script so that the Python files do not need to be altered. If you want it to import from the current working directory use the . notation in bash:
export PYTHONPATH=.
python python_prog.py