I'd like to to install Django into a custom location, I've read the distutils documentation and it suggests that I should be able to do something like the following to install under my home directory (when run from an unpacked django tarball).
> python setup.py install --home=~/code/packages/install --install-purelib=modules --install-platlib=modules --install-scripts=scripts --install-data=data
However, every time I run this, it doesn't seem to concatenate the home path with the separate element paths, and so I simply end up with
modules/
scripts/
data/
In the unpacked tar ball directory. I.e. it seems to be treating modules, scripts etc as simply relative paths to local directory and not relative to the --home specified.
I've tried setting the root with --prefix, and using a setup.cfg and nothing seems to work. --prefix and and --home on their own with no other overrides work, but when used together with --install-xxx overrides it doesn't.
I'm either probably doing something stupid, or the documentation is wrong, or their is a bug. Any help much obliged.
I would strongly suggest that you look at Virtualenv and Pip for creating basically silos of python packages.
The Pinax project uses this exclusively now for bundling requirements together for other people to use, and it's becoming more and more of a defacto standard in the reusable apps space.
Ok, so I've been looking at the distutils source code to see what is going on - distutils.command.install does all of the pathname manipulation.
It turns out that the documentation is actually incorrect. Whenever an --install-xxxx style option is provided it completely overrides any value that might be derived from --home or --prefix - the code not does do any straightforward concatenation of paths.
However, what it does do is variable substitution of a set of special variables. The one of interest to me specifically is $base. Using it on the command line you can define the overrides, and distutils will replace all occurrences with what was specified for --home etc. But note you must quote your filenames so BASH does not try expand it as a environment variable.
So the command line that I had initially, becomes:
python setup.py install --home=/home/andre/code/packages/install --install-purelib='$base/modules' \
--install-platlib='$base/modules' --install-scripts='$base/scripts' --install-data='$base/data'
Hope someone other than me finds that useful!
As a quick check, I'd suggest replacing
~/code/packages/install
with
/full_path_to_your_user/code/packages/install
If you just want it in your home directory, there's no need to install it at all. Just make sure that the container directory is on your pythonpath somewhere, and move the scripts in django/bin into somewhere on your main PATH (or add that dir to your path).
Related
In my office we have a quite complex directory structure when it comes to our code.
One of the things we have is a libs module to drop "common" things used by other parts of our big application (or set of applications... that are all living under a common directory).
The code in that libs/ directory requires certain packages installed in order for it to work. In said libs/ directory we have a requirements.txt file that supposedly lists the dependencies required for the things (things being Python code) in it to work. We have been filling that requirements.txt file pretty manually, tracking that "if this .py file uses this module, we should add it to the requirements file" so it's almost certain that by now we have forgotten adding some required modules.
Because of the complex structure we have (some parts use pipenv, some other have their own requirements.txt...) is very hard knowing whether a required module is going to end up installed or not.
So I would like to make sure that this libs/ directory (cough, cough... module ) has all its dependencies listed in its libs/requirements.txt.
Is that possible? Ideally it'd be "run this command passing /libs/ as an argument, it'll scan the directory and tell you what packages are needed by the py(s) found in it"
Thank you in advance.
Unfortunately, python does not know whether its dependencies are satisfied until runtime. requirements.txt is just a helper file for pip and similar tools, and you have to update it manually.
That said, you could
use the os module to recursively get a list of all *.py files in the folder
parse each one of them for lines having the format import aaa.bbb or from aaa import bbb
keep a set of the imports
However, even in that case, the name of the imported module is not the same as the name you need to pass to pip (eg, import yaml requires pyyaml in requirements.txt), but at least it could be a hint of what's missing.
Edit: based on a Ulf Rompe's comment, it is important you use "1" instead of "0", otherwise you will break sys.path.
I have been doing python for quite a while now (over a year), and I am always confused as to why people recommend you use sys.path.append() instead of sys.path.insert(). Let me demonstrate.
Let's say I am working on a module named PyWorkbooks (that is installed on my computer), but I am simultaneously working on a different module (let's say PyJob) that incorporates PyWorkbooks. As I'm working on PyJob I find errors in PyWorkbooks that I am correcting, so I would like to import a development version.
There are multiple ways to work on both (I could put my PyWorkbooks project inside of PyJob, for instance), but sometimes I will still need to play with the path. However, I cannot simply do a sys.path.append() to the folder where PyWorkbooks is at. Why? Because python will find my installed PyWorkbooks first!
This is why you have to do a sys.path.insert(1, path_to_dev_pyworkbooks)
In summary:
sys.path.append(path_to_dev_pyworkbooks)
import PyWorkbooks # does NOT import dev pyworkbooks, imports installed one
or:
sys.path.insert(1, path_to_dev_pyworkbooks) # based on comments you should use **1 not 0**
import PyWorkbooks # imports correct file
This has caused a few hangups for me in the past, and I would really like it if we (as a community) started recommending sys.path.insert(1, path), as if you are manually inserting a path I think it is safe to say that that is the path you want to use!
Or do I have something wrong? It's a question that sometimes bothers me and I wanted it in the open!
If you really need to use sys.path.insert, consider leaving sys.path[0] as it is:
sys.path.insert(1, path_to_dev_pyworkbooks)
This could be important since 3rd party code may rely on sys.path documentation conformance:
As initialized upon program startup, the first item of this list,
path[0], is the directory containing the script that was used to
invoke the Python interpreter.
If you have multiple versions of a package / module, you need to be using virtualenv (emphasis mine):
virtualenv is a tool to create isolated Python environments.
The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.
Or more generally, what if you want to install an application and leave it be? If an application works, any change in its libraries or the versions of those libraries can break the application.
Also, what if you can’t install packages into the global site-packages directory? For instance, on a shared host.
In all these cases, virtualenv can help you. It creates an environment that has its own installation directories, that doesn’t share libraries with other virtualenv environments (and optionally doesn’t access the globally installed libraries either).
That's why people consider insert(0, to be wrong -- it's an incomplete, stopgap solution to the problem of managing multiple environments.
you are confusing the concept of appending and prepending. the following code is prepending:
sys.path.insert(1,'/thePathToYourFolder/')
it places the new information at the beginning (well, second, to be precise) of the search sequence that your interpreter will go through. sys.path.append() puts things at the very end of the search sequence.
it is advisable that you use something like virtualenv instead of manually coding your package directories into the PYTHONPATH everytime. for setting up various ecosystems that separate your site-packages and possible versions of python, read these two blogs:
python ecosystems introduction
bootstrapping python virtual environments
if you do decide to move down the path to environment isolation you would certainly benefit by looking into virtualenvwrapper: http://www.doughellmann.com/docs/virtualenvwrapper/
I need to put a python script somewhere on my computer so that in another file I can use it. How do I do this and where do I put it? And where in the python documentation do I learn how to do this? I'm a beginner + don't use python much.
library file: MyLib.py put in a well-known place
def myfunc():
....
other file SourceFile.py located elsewhere, doesn't need to know where MyLib.py is:
something = MyLib.myfunc()
Option 1:
Put your file at:
<Wherever your Python is>/Lib/site-packages/myfile.py
Add this to your code:
import myfile
Pros: Easy
Cons: Clutters site-packages
Option 2:
Put your file at:
/Lib/site-packages/mypackage/myfile.py
Create an empty text file called:
<Wherever your Python is>/Lib/site-packages/mypackage/__init__.py
Add this to your code:
from mypackage import myfile
Pros: Reduces clutter in site-packages by keeping your stuff consolidated in a single directory
Cons: Slightly more work; still some clutter in site-packages. This isn't bad for stable stuff, but may be regarded as inappropriate for development work, and may be impossible if Python is installed on a shared drive
Option 3
Put your file in any directory you like
Add that directory to the PYTHONPATH environment variable
Proceed as with Option 1 or Option 2, except substitute the directory you just created for <Wherever your Python is>/Lib/site-packages/
Pros: Keeps development code out of the site-packages directory
Cons: slightly more setup
This is the approach I usually use for development work
In general, the Modules section of the Python tutorial is a good introduction for beginners on this topic. It explains how to write your own modules and where to put them, but I'll summarize the answer to your question below:
Your Python installation has a site-packages directory; any python file you put in that directory will be available to any script you write. For example, if you put the file MyLib.py in the site-packages directory, then in your script you can say
import MyLib
something = MyLib.myfunc()
If you're not sure where Python is installed, the Stack Overflow question How do I find the location of my Python site-packages directory will be helpful to you.
Alternatively, you can modify sys.path, which is a list of directories where Python looks for libraries when you use the import statement. Your site-packages directory is already in this list, but you can add (or remove) entries yourself. For example, if you wanted to put your MyLib.py file in /usr/local/pythonModules, you could say
import sys
sys.path.append("/usr/local/pythonModules")
import MyLib
something = MyLib.myfunc()
Finally, you could use the PYTHONPATH environment variable to indicate the directory where your MyLib.py is located.
However, I recommend simply placing your MyLib.py file in the site-packages directory, as described above.
No one has mentioned using .pth files in site-packages to abstract away the location.
You will have to place your MyLib.py somewhere in your load path (this the paths in your sys.path variable) and then you'll be able to import it fine. Your code would look like
import MyLib
MyLib.myfunc()
Generally speaking, you should distribute your packages using distutils so that they can be easily installed in the proper locations. It would help you as well.
Also, you might not want to install packages in your global Python install. It's customary (and recommended) to use virtualenv which you can use to create small isolated Python environments that can hold local packages.
It's best your give the whole thing a shot and then ask further questions if you have them.
The private version, from my .profile
export PYTHONPATH=${PYTHONPATH}:$HOME/lib/python
which has a subdirectory "msw" so import msw.primes is self documenting or add to a local directory that is already in sys.path
The Python tutorial section 6 talks about modules, and 6.1.2 talks about the PYTHONPATH, which determines where Python will look for modules you try to import. The tutorial: http://docs.python.org/tutorial/modules.html
Lets say I have projects x and y in brother directories: projects/x and projects/y.
There are some utility funcs common to both projects in myutils.py and some db stuff in mydbstuff.py, etc.
Those are minor common goodies, so I don't want to create a single package for them.
Questions arise about the whereabouts of such files, possible changes to PYTHONPATH, proper way to import, etc.
What is the 'pythonic way' to use such files?
The pythonic way is to create a single extra package for them.
Why don't you want to create a package? You can distribute this package with both projects, and the effect would be the same.
You'll never do it right for all instalation scenarios and platforms if you do it by mangling with PYTHONPATH and custom imports.
Just create another package and be done in no time.
You can add path to shared files to sys.path either directly by sys.path.append(pathToShared) or by defining .pth files and add them to with site.addsitedir. Path files (.pth) are simple text files with a path in each line.
You can also create a .pth file, which will store the directory(ies) that you want added to your PYTHONPATH. .pth files are copied to the Python/lib/site-packages directory, and any directory in that file will be added to your PYTHONPATH at runtime.
http://docs.python.org/library/site.html
StackOVerflow question (see accepted solution)
I agree with 'create a package'.
If you cannot do that, how about using symbolic links/junctions (ln -s on Linux, linkd on Windows)?
I'd advise using setuptools for this. It allows you to set dependencies so you can make sure all of these packages/individual modules are on the sys.path before installing a package. If you want to install something that's just a single source file, it has support for automagically generating a simple setup.py for it. This may be useful if you decide not to go the package route.
If you plan on deploying this on multiple computers, I will usually set up a webserver with all the dependencies I plan on using so it can install them for you automatically.
I've also heard good things about paver, but haven't used it myself.
I'm in the middle of reworking our build scripts to be based upon the wonderful Waf tool (I did use SCons for ages but its just way too slow).
Anyway, I've hit the following situation and I cannot find a resolution to it:
I have a product that depends on a number of previously built egg files.
I'm trying to package the product using PyInstaller as part of the build process.
I build the dependencies first.
Next I want to run PyInstaller to package the product that depends on the eggs I built. I need PyInstaller to be able to load those egg files as part of it's packaging process.
This sounds easy: you work out what PYTHONPATH should be, construct a copy of sys.environ setting the variable up correctly, and then invoke the PyInstaller script using subprocess.Popen passing the previously configured environment as the env argument.
The problem is that setting PYTHONPATH alone does not seem to be enough if the eggs you are adding are extension modules that are packaged as zipsafe. In this case, it turns out that the embedded libraries are not able to be imported.
If I unzip the eggs (renaming the directories to .egg), I can import them with no further settings but this is not what I want in this case.
I can also get the eggs to import from a subshell by doing the following:
Setting PYTHONPATH to the directory that contains the egg you want to import (not the path of the egg itself)
Loading a python shell and using pkg_resources.require to locate the egg.
Once this has been done, the egg loads as normal. Again, this is not practical because I need to be able to run my python shell in a manner where it is ready to import these eggs from the off.
The dirty option would be to output a wrapper script that took the above actions before calling the real target script but this seems like the wrong thing to do: there must be a better way to do this.
Heh, I think this was my bad. The issue appear to have been that the zipsafe flag in setup.py for the extension package was set to False, which appears to affect your ability to treat it as such at all.
Now that I've set that to True I can import the egg files, simply by adding each one to the PYTHONPATH.
I hope someone else finds this answer useful one day!
Although you have a solution, you could always try "virtualenv" that creates a virtual environment of python where you can install and test Python Packages without messing with the core system python:
http://pypi.python.org/pypi/virtualenv