I am pretty new to Python so it might sound obvious but I haven't found this everywhere else
Say I have a application (module) in the directoy A/ then I start developing an application/module in another directory B/
So right now I have
source/
|_A/
|_B/
From B I want to use functions are classes defined in B. I might eventually pull them out and put them in a "misc" or "util" module.
In any case, what is the best way too add to the PYTHONPATH module B so A can see it? taking into account that I will be also making changes to B.
So far I came up with something like:
def setup_paths():
import sys
sys.path.append('../B')
when I want to develop something in A that uses B but this just does not feel right.
Normally when you are developing a single application your directory structure will be similar to
src/
|-myapp/
|-pkg_a/
|-__init__.py
|-foo.py
|-pkg_b/
|-__init__.py
|-bar.py
|-myapp.py
This lets your whole project be reused as a package by others. In myapp.py you will typically have a short main function.
You can import other modules of your application easily. For example, in pkg_b/bar.py you might have
import myapp.pkg_a.foo
I think it's the preferred way of organising your imports.
You can do relative imports if you really want, they are described in PEP-328.
import ..pkg_a.foo
but personally I think, they are a bit ugly and difficult to maintain (that's arguable, of course).
Of course, if one of your modules needs a module from another application it's a completely different story, since this application is an external dependency and you'll have to handle it.
I would recommend using the imp module
import imp
imp.load_source('module','../B/module.py')
Else use absolute path starting from root
def setup_paths():
import sys
sys.path.append('/path/to/B')
Add proper folder to PYTHONPATH
First, create the root directory for package that you create. Let's call it project.
Subdirs of project are your apps, that is A and B. Now you have to add the parent directory of project to PYTHONPATH.
Refering modules inside the package.
Let's say you have other_app.py in B, so in A/app_name.py you import it like this:
from project.B.other_app import *
Or if you want to have all symbols in other_app namespace import like this:
from project.B import other_app
Nice way of creating launchers and dynamically change PYTHONPATH
If you want to create universal app launchers for Python which work even when you move your package to other PC/dir you need some solution to dynamically add package parent directory to PYTHONPATH. Here is my solution to this case (for Linux, but you can also translate simple scripts to Windows if you google a bit :) )
In the same folder in which you created project create pwd_to_pythonpath.sh, short bash script:
#!/bin/bash
export PYTHONPATH=$PYTHONPATH:`pwd`
Then create launcher for your A and B, for example A.sh would look like:
#!/bin/bash
source pwd_to_pythonpath.sh
python -m project.A.app_name
The app_name should be the same as module file name in A folder (app_name.py in this case)
Related
In putting together a convolutional model in my computer, I want to use the convert_to_one_hot(.. utility written by Coursera/Ng. I can download their module test_utils.py, where I expect the one_hot converter to be. How do I incorporate test_utils.py to my tools? with Import? Install? where do I put the test_utils.py, or similar collections?
Im guessing you are referring to some built in code that Andrew Ng gave, in that case if you know its a single file you can just put it wherever you want in your project and make sure that you reference to the nearest _init_.py inside your project, so for example, imagine your project looks something like this:
YourProject
app.py
my_module
__init__.py
test_utils.py
config.py
If you were working on the app.py file and wanted to use the test_utils.py function called one_hot_encoder(), you could simply do
from my_module.test_utils import one_hot_encoder
And thats because there is a _init_.py file inside the my_module folder, which the python interpreter take that folder as a "library" name and searchs for the root you are specifying (test_utils) and returns whatever import you choose, could be either a class or a function or even a variable like a list.
NOTE: Make sure that you add other necessary scripts to your module or to pip install the libraries used in the script, for it to work.
I have read how pythons module system works, but it makes things IMO too formal because I must access everything from some script in the root package which makes things a little harder to deal with IMO. I am not making packages I want to publish with a nice API, but just loose conglomerations of scripts for my own research...
So I want to have everything work by being able to relative import things from parent directories (or maybe parents of parents) and still be able to call scripts from within the child directories.
project/
dir/
child/
script1.py
dir2/
script2.py
utils/
utils.py
I want to be able to import utils from script1.py and be able to directly call script1.py without having to set up a traditional project. Is there a way to do this?
Actually you can add whatever you want to environment variable PYTHONPATH. All the modules under the path can be accessed. For example, in your snippet, you can set PYTHONPATH=/path/to/project, then you can access modules by import utils.tuils or import dir2.script2.py. Somehow like this.
I have just finished writing the core section of a project I am working on and I want to write test for it using unittest before I continue. I am aware that I should have done this before, but when I started I didn't know Python, so.. whatever..
What I would like to achieve: I have a sub-package of the main package which contains all the modules I want to test inside it. I want to put a subsubpackage inside that called 'tests' or something which then contains all of my test cases, which I'd like to be able to aggregate into a test suite from outside the package so eventually I can run all the test for the entire project in one go.
The structure is something like this:
/projectPackage
/projectPackage/package
/projectPackage/package/\__init__.py (empty)
/projectPackage/package/someModule.py
/projectPackage/package/... (more modules)
/projectPackage/package/testing.py (runs all the tests in /tests/)
/projectPackage/package/tests
/projectPackage/package/tests/\__init__.py (empty)
/projectPackage/package/tests/someModuleTests.py
Problem I am having:
someModuleTests has to import someModule from the parent package so it can test its methods. This doesn't seem to work. I get various errors like:
Attempted relative import beyond toplevel package
Anyway, I expect it's just because I am a Python noob! I have my own ideas on how I am going to do it for this project, because of course each is different, but any general advice on the structuring of medium-large python projects is also appreciated.
Run the unit test from the parent directory so the directory is in your PYTHONPATH (the current working directory always is). This is done by executing the test file from your parent directory or by using something like nosetest which recursively looks for all tests in your package.
Don't use relative imports, they cause things like this. Don't hack your PYTHONPATH and/or sys.path to try and fix it either.
Considering that you are using command line tools to run your tests, you can follow the docs and have something like the following:
python -m unittest package.tests.someModuleTests
If you can execute your code then your PYTHONPATH should already be set correctly. What you need to do is to specify the module you want to import, e.g.
import projectPackage.package.someModule
I need to put a python script somewhere on my computer so that in another file I can use it. How do I do this and where do I put it? And where in the python documentation do I learn how to do this? I'm a beginner + don't use python much.
library file: MyLib.py put in a well-known place
def myfunc():
....
other file SourceFile.py located elsewhere, doesn't need to know where MyLib.py is:
something = MyLib.myfunc()
Option 1:
Put your file at:
<Wherever your Python is>/Lib/site-packages/myfile.py
Add this to your code:
import myfile
Pros: Easy
Cons: Clutters site-packages
Option 2:
Put your file at:
/Lib/site-packages/mypackage/myfile.py
Create an empty text file called:
<Wherever your Python is>/Lib/site-packages/mypackage/__init__.py
Add this to your code:
from mypackage import myfile
Pros: Reduces clutter in site-packages by keeping your stuff consolidated in a single directory
Cons: Slightly more work; still some clutter in site-packages. This isn't bad for stable stuff, but may be regarded as inappropriate for development work, and may be impossible if Python is installed on a shared drive
Option 3
Put your file in any directory you like
Add that directory to the PYTHONPATH environment variable
Proceed as with Option 1 or Option 2, except substitute the directory you just created for <Wherever your Python is>/Lib/site-packages/
Pros: Keeps development code out of the site-packages directory
Cons: slightly more setup
This is the approach I usually use for development work
In general, the Modules section of the Python tutorial is a good introduction for beginners on this topic. It explains how to write your own modules and where to put them, but I'll summarize the answer to your question below:
Your Python installation has a site-packages directory; any python file you put in that directory will be available to any script you write. For example, if you put the file MyLib.py in the site-packages directory, then in your script you can say
import MyLib
something = MyLib.myfunc()
If you're not sure where Python is installed, the Stack Overflow question How do I find the location of my Python site-packages directory will be helpful to you.
Alternatively, you can modify sys.path, which is a list of directories where Python looks for libraries when you use the import statement. Your site-packages directory is already in this list, but you can add (or remove) entries yourself. For example, if you wanted to put your MyLib.py file in /usr/local/pythonModules, you could say
import sys
sys.path.append("/usr/local/pythonModules")
import MyLib
something = MyLib.myfunc()
Finally, you could use the PYTHONPATH environment variable to indicate the directory where your MyLib.py is located.
However, I recommend simply placing your MyLib.py file in the site-packages directory, as described above.
No one has mentioned using .pth files in site-packages to abstract away the location.
You will have to place your MyLib.py somewhere in your load path (this the paths in your sys.path variable) and then you'll be able to import it fine. Your code would look like
import MyLib
MyLib.myfunc()
Generally speaking, you should distribute your packages using distutils so that they can be easily installed in the proper locations. It would help you as well.
Also, you might not want to install packages in your global Python install. It's customary (and recommended) to use virtualenv which you can use to create small isolated Python environments that can hold local packages.
It's best your give the whole thing a shot and then ask further questions if you have them.
The private version, from my .profile
export PYTHONPATH=${PYTHONPATH}:$HOME/lib/python
which has a subdirectory "msw" so import msw.primes is self documenting or add to a local directory that is already in sys.path
The Python tutorial section 6 talks about modules, and 6.1.2 talks about the PYTHONPATH, which determines where Python will look for modules you try to import. The tutorial: http://docs.python.org/tutorial/modules.html
I'm just beginning Python, and I'd like to use an external RSS class. Where do I put that class and how do I import it? I'd like to eventually be able to share python programs.
About the import statement:
(a good writeup is at http://effbot.org/zone/import-confusion.htm and the python tutorial goes into detail at http://docs.python.org/tutorial/modules.html )
There are two normal ways to import code into a python program.
Modules
Packages
A module is simply a file that ends in .py. In order for python, it must exist on the search path (as defined in sys.path). The search path usually consists of the same directory of the .py that is being run, as well as the python system directories.
Given the following directory structure:
myprogram/main.py
myprogram/rss.py
From main.py, you can "import" the rss classes by running:
import rss
rss.rss_class()
#alternativly you can use:
from rss import rss_class
rss_class()
Packages provide a more structured way to contain larger python programs. They are simply a directory which contains an __init__.py as well as other python files.
As long as the package directory is on sys.path, then it can be used exactly the same as above.
To find your current path, run this:
import sys
print(sys.path)
I don't really like answering so late, but I'm not entirely satisfied with the existing answers.
I'm just beginning Python, and I'd like to use an external RSS class. Where do I put that class and how do I import it?
You put it in a python file, and give the python file an extension of .py . Then you can import a module representing that file, and access the class. Supposing you want to import it, you must put the python file somewhere in your import search path-- you can see this at run-time with sys.path, and possibly the most significant thing to know is that the site-packages (install-specific) and current directory ('') are generally in the import search path. When you have a single homogeneous project, you generally put it in the same directory as your other modules and let them import each other from the same directory.
I'd like to eventually be able to share python programs.
After you have it set up as a standalone file, you can get it set up for distribution using distutils. That way you don't have to worry about where, exactly, it should be installed-- distutils will worry for you. There are many other additional means of distribution as well, many OS-specific-- distutils works for modules, but if you want to distribute a proper program that users are meant to run, other options exist, such as using py2exe for Windows.
As for the modules/packages distinction, well, here it goes. If you've got a whole bunch of classes that you want divided up so that you don't have one big mess of a python file, you can separate it into multiple python files in a directory, and give the directory an __init__.py . The important thing to note is that from Python, there's no difference between a package and any other module. A package is a module, it's just a different way of representing one on the filesystem. Similarly, a module is not just a .py file-- if that were the case, sys would not be a module, since it has no .py file. It's built-in to the interpreter. There are infinitely many ways to represent modules on the filesystem, since you can add import hooks that can create ways other than directories and .py files to represent modules. One could, hypothetically, create an import hook that used spidermonkey to load Javascript files as Python modules.
from [module] import [classname]
Where the module is somewhere on your python path.
About modules and packages:
a module is a file ending with .py. You can put your class in such a file. As said by Andy, it needs to be in your python path (PYTHONPATH). Usually you will put the additional module in the same directory as your script is though which can be directly imported.
a package is a directory containing an __init__.py (can be empty) and contains module files. You can then import a la from <package>.<module> import <class>. Again this needs to be on your python path.
You can find more in the documenation.
If you want to store your RSS file in a different place use sys.append("") and pout the module in that directory and use
import or from import *
The first file, where you have created the class, is "first.py"
first.py:
class Example:
...
You create the second file, where you want to use the class contained in the "first.py", which is "second.py"
myprogram/first.py
myprogram/second.py
Then in the second file, to call the class contained in the first file, you simply type:
second.py:
from first import Example
...