What would be the best directory structure strategy to share a utilities module across my python projects? As the common modules would be updated with new functions I would not want to put them in the python install directory.
project1/
project2/
sharedUtils/
From project1 I can not use "import ..\sharedUtils", is there any other way? I would rather not hardcode the "sharedUtils" location
Thanks in advance
Directory structure:
project1/foo.py
sharedUtils/bar.py
With the directories as you've shown them, from foo.py inside the project1 directory you can add the relative path to sharedUtils as follows:
import sys
sys.path.append("../sharedUtils")
import bar
This avoids hardcoding a C:/../sharedUtils path, and will work as long as you don't change the directory structure.
Make a separate standalone package? And put it in the /site-packages of your python install?
There is also my personal favorite when it comes to development mode: use of symlinks and/or *.pth files.
Suppose you have sharedUtils/utils_foo and sharedUtils/utils_bar.
You could edit your PYTHONPATH to include sharedUtils, then import them in project1 and project2 using
import utils_foo
import utils_bar
etc.
In linux you could do that be editing ~/.profile with something like this:
PYTHONPATH=/path/to/sharedUtils:/other/paths
export PYTHONPATH
Using the PYTHONPATH environment variable affects the directories that python searches when looking for modules. Since every user can set his own PYTHONPATH, this solution is good for personal projects.
If you want all users on the machine to be able to import modules in sharedUtils, then
you can achieve this by using a .pth file. Exactly where you put the .pth file may depend on your python distribution. See Using .pth files for Python development.
Related
I'm trying to set up an environment for Python project to help out a friend (developing on different OS).
I want to set environment variables per project in the .env file, but ultimately it fails to recognize Pythonpath.
The goal is to set the /api subfolder of my currently opened folder, but ir doesn't recognize the relative path. Writing there absolute path (as in screenshot below) sort of works - it recognizes the custom library and all others, but on debugging it fails on other relative paths in the code (or so it appears)
Any advice? Thanks.
One approach is to avoid the whole environment setup to begin with and define your dependencies in source. This works when your repo has all you need.
Consider the project structure like this
proj
/api
api_source.py
/utils
utils_source.py
/tests
api_source.py
import os
import sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../')))
import utils.utils_source as utils_source
It's a bit messy but I like this approach in that I do not have to setup any environment wherever I clone my repo, at least not for the self contained dependencies within the repo itself.
Usually I'll do this path appending in the Main script and structure all my imports in the dependencies to rely on a fixed path prefix in the path list that way I do this operation once and all the dependent modules run with it.
I'm climbing my learning curve in Python and try to understand where to put everything.
I originally have a python module in a folder and then a sub folder src, in this src folder I will then have my main source files say main.py then I will have models folder storing my models codes.
/myproject/src/main.py
/myproject/src/models/a-model.py
/myproject/src/models/b-model.py
So my main will import the model like this:
from models.a-model import a
Then when I package the zip file I just zip the myproject folder with that folder structure and deploy and everything is fine.
Now I have another new module doing something different but need to use the same models.
I can easily duplicate them all and code separately and deploy. But I would like to share the codes to the models, so that when one model changes, I only need to update once, instead of 2 places.
My new module is like
/mynew/src/main-b.py
/mynew/src/models/a-model.py
/mynew/src/models/b-model.py
What is the best practise to do this?
Do I put like this?
/myproject/src/main.py
/mynew/src/main-b.py
/models/a-model.py
/models/b-model.py
And then update the import?
But I have doubt how do I deploy? Do I also have to setup the same folder structures?
One would be adding /myproject/src/models to the environment variable PYTHONPATH. Python adds the directories listed in PYTHONPATH environment variable to sys.path, the list of directories where Python searches when you try to import something. This is bad, because modifying PYTHONPATH has its own side effects, fortunately, virtual environments provide a way to get around those side effects.
Alternatively and much better you could add your modules to site-packages directory, site-packages is added to sys.pathby default, this obviates the need to modifyPYTHONPATH. To locate thesite-packages` directory, refer to this page from Python Documentation: Installing Python Modules (Legacy version).
You could also use LiClipse IDE which comes with Pydev already installed. Create source a folder from the IDE and link your previous project with your newer project. When you link your projects the IDE adds the source folders of your older project to the PYTHONPATH of your newer project and thus Python will be able to locate your modules.
I am pretty new to Python so it might sound obvious but I haven't found this everywhere else
Say I have a application (module) in the directoy A/ then I start developing an application/module in another directory B/
So right now I have
source/
|_A/
|_B/
From B I want to use functions are classes defined in B. I might eventually pull them out and put them in a "misc" or "util" module.
In any case, what is the best way too add to the PYTHONPATH module B so A can see it? taking into account that I will be also making changes to B.
So far I came up with something like:
def setup_paths():
import sys
sys.path.append('../B')
when I want to develop something in A that uses B but this just does not feel right.
Normally when you are developing a single application your directory structure will be similar to
src/
|-myapp/
|-pkg_a/
|-__init__.py
|-foo.py
|-pkg_b/
|-__init__.py
|-bar.py
|-myapp.py
This lets your whole project be reused as a package by others. In myapp.py you will typically have a short main function.
You can import other modules of your application easily. For example, in pkg_b/bar.py you might have
import myapp.pkg_a.foo
I think it's the preferred way of organising your imports.
You can do relative imports if you really want, they are described in PEP-328.
import ..pkg_a.foo
but personally I think, they are a bit ugly and difficult to maintain (that's arguable, of course).
Of course, if one of your modules needs a module from another application it's a completely different story, since this application is an external dependency and you'll have to handle it.
I would recommend using the imp module
import imp
imp.load_source('module','../B/module.py')
Else use absolute path starting from root
def setup_paths():
import sys
sys.path.append('/path/to/B')
Add proper folder to PYTHONPATH
First, create the root directory for package that you create. Let's call it project.
Subdirs of project are your apps, that is A and B. Now you have to add the parent directory of project to PYTHONPATH.
Refering modules inside the package.
Let's say you have other_app.py in B, so in A/app_name.py you import it like this:
from project.B.other_app import *
Or if you want to have all symbols in other_app namespace import like this:
from project.B import other_app
Nice way of creating launchers and dynamically change PYTHONPATH
If you want to create universal app launchers for Python which work even when you move your package to other PC/dir you need some solution to dynamically add package parent directory to PYTHONPATH. Here is my solution to this case (for Linux, but you can also translate simple scripts to Windows if you google a bit :) )
In the same folder in which you created project create pwd_to_pythonpath.sh, short bash script:
#!/bin/bash
export PYTHONPATH=$PYTHONPATH:`pwd`
Then create launcher for your A and B, for example A.sh would look like:
#!/bin/bash
source pwd_to_pythonpath.sh
python -m project.A.app_name
The app_name should be the same as module file name in A folder (app_name.py in this case)
First off, I'm pretty new so I hope I hadn't missed anything too trivial.
Here's a small preface with lots of info:
I'm using Windows & Python 2.7.
I've been using an open-source module named pybrain, which I need to change pretty much for my own purposes. So far I've been changing it directly from the python site-packages folder, but I guess it's a pretty messy way to work, so I decided to try and re-do thing so as to launch it from a different folder.
I've also decided to start using Aptana (which as far as I can gather is Eclipse-based enough for the same solutions to apply) instead of the messier but simpler "Spyder" I've been using so far.
Pybrain is a pretty layered module with lots of different subfolders, e.g.:
pybrain
--> subfolder1
--> subfolder2
...
So far I've figured these out:
- I've removed the path to the pybrain folder in site-packages from the PYTHONPATH in aptana project.
- I've added the path to the new project folder.
This works for some imports, namely, the ones that only reference relative paths inside the subfolders, e.g. I can import from things in subfolder1 if I write a module in the main folder.
However, whenever I try to import things from the other subfolder - I can't use "pybrain" in the hierarchy:
from pybrain.subfolder2 import *
doesn't work in subfolder1.
And here is my question:
How do I configure "pybrain" to be a usable name in the code, just as it was when I had pybrain in the site-packages folder?
I think you have added the wrong path to your source folder...
I.e.:
if you have a structure
/project
/project/pybrain
/project/pybrain/__init__.py
The source folder set should be '/project' (whereas I think you've set /project/pybrain as the source folder)... if that's not the case, please add more information on your folders and what did you set as a source folder...
Probably Aptana has some way to configure the list of folders which are considered source packages as in pycharm and in eclipse-pydev.
In any case, you could access your module using a .pth file in your site-packages. This file can be named as you want (p.e. pybrain.pth) and should contain only one line with the path to your pybrain folder. See this and this.
I need to put a python script somewhere on my computer so that in another file I can use it. How do I do this and where do I put it? And where in the python documentation do I learn how to do this? I'm a beginner + don't use python much.
library file: MyLib.py put in a well-known place
def myfunc():
....
other file SourceFile.py located elsewhere, doesn't need to know where MyLib.py is:
something = MyLib.myfunc()
Option 1:
Put your file at:
<Wherever your Python is>/Lib/site-packages/myfile.py
Add this to your code:
import myfile
Pros: Easy
Cons: Clutters site-packages
Option 2:
Put your file at:
/Lib/site-packages/mypackage/myfile.py
Create an empty text file called:
<Wherever your Python is>/Lib/site-packages/mypackage/__init__.py
Add this to your code:
from mypackage import myfile
Pros: Reduces clutter in site-packages by keeping your stuff consolidated in a single directory
Cons: Slightly more work; still some clutter in site-packages. This isn't bad for stable stuff, but may be regarded as inappropriate for development work, and may be impossible if Python is installed on a shared drive
Option 3
Put your file in any directory you like
Add that directory to the PYTHONPATH environment variable
Proceed as with Option 1 or Option 2, except substitute the directory you just created for <Wherever your Python is>/Lib/site-packages/
Pros: Keeps development code out of the site-packages directory
Cons: slightly more setup
This is the approach I usually use for development work
In general, the Modules section of the Python tutorial is a good introduction for beginners on this topic. It explains how to write your own modules and where to put them, but I'll summarize the answer to your question below:
Your Python installation has a site-packages directory; any python file you put in that directory will be available to any script you write. For example, if you put the file MyLib.py in the site-packages directory, then in your script you can say
import MyLib
something = MyLib.myfunc()
If you're not sure where Python is installed, the Stack Overflow question How do I find the location of my Python site-packages directory will be helpful to you.
Alternatively, you can modify sys.path, which is a list of directories where Python looks for libraries when you use the import statement. Your site-packages directory is already in this list, but you can add (or remove) entries yourself. For example, if you wanted to put your MyLib.py file in /usr/local/pythonModules, you could say
import sys
sys.path.append("/usr/local/pythonModules")
import MyLib
something = MyLib.myfunc()
Finally, you could use the PYTHONPATH environment variable to indicate the directory where your MyLib.py is located.
However, I recommend simply placing your MyLib.py file in the site-packages directory, as described above.
No one has mentioned using .pth files in site-packages to abstract away the location.
You will have to place your MyLib.py somewhere in your load path (this the paths in your sys.path variable) and then you'll be able to import it fine. Your code would look like
import MyLib
MyLib.myfunc()
Generally speaking, you should distribute your packages using distutils so that they can be easily installed in the proper locations. It would help you as well.
Also, you might not want to install packages in your global Python install. It's customary (and recommended) to use virtualenv which you can use to create small isolated Python environments that can hold local packages.
It's best your give the whole thing a shot and then ask further questions if you have them.
The private version, from my .profile
export PYTHONPATH=${PYTHONPATH}:$HOME/lib/python
which has a subdirectory "msw" so import msw.primes is self documenting or add to a local directory that is already in sys.path
The Python tutorial section 6 talks about modules, and 6.1.2 talks about the PYTHONPATH, which determines where Python will look for modules you try to import. The tutorial: http://docs.python.org/tutorial/modules.html