I have a python project pypypy with 2 files: __main__.py and foo.py.
In __main__.py I simply do import via import foo. It all works fine.
Now, I want to distribute it with pypi. After installing my module I'm execution it with python -m pypypy. When I do that, the import statement doesn't work anymore. However import pypypy.foo does the job.
Should I change all my imports before distribution or there is a better way?
Using absolute imports is strongly suggested as they work consistently across different python versions. Check this answer. In your case you should prefer using import pypypy.foo.
The reason it works in your dev environment might be because of PYTHONPATH manipulation. For example Pycharm automatically sets Add content and source roots to PYTHONPATH. Also when you run python it automatically adds current working directory to PYTHONPATH.
It's been a while that I am struggling with imports in packages. When I develop a package, I read everywhere that it is preferable to use absolute imports in submodules of that package. I understand that and I like it more as well. But then I don't like and I also read that you shouldn't use sys.path.append('/path/to/package') to use your package in development...
So my question is, how do you develop such a package from zero, using directly absolute imports? At the moment I develop the package using relative imports, since then I am able to test the code I am writing before packaging and installing, then I change the imports once I have a release and build the package.
What is the correct way of doing such thing? In Pycharm for example you would mark the folder as 'source roor' and be able to work as if the package folder was in the path. Still I read that this is not the proper way... what am I missing? How do you develop a package while testing its code?
Your mileage may vary but this is what I usually do:
Within a package (foo), absolute (import foo.bar) or relative (import .bar) doesn't matter to me as long as it works. Sometimes, I prefer relative especially when the project is large and one day I might decide to move a number of source files into a subdirectory.
How do I test? My $PYTHONPATH usually has . in it, and my directory hierarchy is like this:
/path/to/foo_project
/setup.py
/foo
/__init__.py
/bar.py
/test
/test1.py
/test2.py
then the script in foo_project/test/test1.py will be like what you normally use the package, using import foo.bar. And when I test my code, I will be in the directory foo_project and run python test/test1.py. Since I have . in my $PYTHONPATH, it will find the directory foo and use it as a package.
I have a module that I want to keep up to date, and I'm wondering if this is a bad idea:
Have a module (mod1.py) in the
site-packages directory that copies a
different module from some other
location into the site-packages
directory, and then imports * from
that module.
import shutil
from distutils.sysconfig import get_python_lib
p_source = r'\\SourceSafeServer\mod1_current.py'
p_local = get_python_lib() + r'\mod1_current.py'
shutil.copyfile(p_source, p_local)
from mod1_current import *
Now I can do this in any module, and it will always be the latest version:
from mod1 import function1
This works.... but is there a better way of doing this?
Update
Here is the current process... there is a project under source-control that has a single module: mod1.py There is also a setup.py Running setup.py copies mod1.py to the site-packages directory.
Developers that use the module must run setup.py to update the module. Sometimes, they don't and not having the latest version causes problems.
I want to be able to just check-in the a new version, and any code that imports that module will automatically grab the latest version every time, without anyone having to run setup.py
Do you really want to do this? This means you could very easily roll code to a production app simply by committing to source control. I would consider this a nasty side-effect for someone who isn't aware of your setup.
That being said this seems like a pretty good solution - you may want to add some exception-handling around the network file calls as those are prone to failure.
In some cases, we put .pth files in the Python site-packages directory. The .pth files name our various SVN checkout directories.
No install. No copy.
.pth files are described here.
The original strategy of having other developers copy mod1.py into their site-packages in order to use the module sounds like it's the real problem. Why aren't they just using the same source control are you are?
This auto-copying will make it hard to do rollbacks, especially if other developers copy your strategy. Imagine this same system used for dozens and dozens of files. And then imagine you actually do want to use a version of mod1.py that is not the latest for something.
I'm thinking how to arrange a deployed python application which will have a
Executable script located in /usr/bin/ which will provide a CLI to functionality implemented in
A library installed to wherever the current site-packages directory is.
Now, currently, I have the following directory structure in my sources:
foo.py
foo/
__init__.py
...
which I guess is not the best way to do things. During development, everything works as expected, however when deployed, the "from foo import FooObject" code in foo.py seemingly attempts to import foo.py itself, which is not the behaviour I'm looking for.
So the question is what is the standard practice of orchestrating situations like this? One of the things I could think of is, when installing, rename foo.py to just foo, which stops it from importing itself, but that seems rather awkward...
Another part of the problem, I suppose, is that it's a naming challenge. Perhaps call the executable script foo-bin.py?
This article is pretty good, and shows you a good way to do it. The second item from the Do list answers your question.
shameless copy paste:
Filesystem structure of a Python project
by Jp Calderone
Do:
name the directory something related to your project. For example, if your
project is named "Twisted", name the
top-level directory for its source
files Twisted. When you do releases,
you should include a version number
suffix: Twisted-2.5.
create a directory Twisted/bin and put your executables there, if you
have any. Don't give them a .py
extension, even if they are Python
source files. Don't put any code in
them except an import of and call to a
main function defined somewhere else
in your projects.
If your project is expressable as a single Python source file, then put it
into the directory and name it
something related to your project. For
example, Twisted/twisted.py. If you
need multiple source files, create a
package instead (Twisted/twisted/,
with an empty
Twisted/twisted/__init__.py) and place
your source files in it. For example,
Twisted/twisted/internet.py.
put your unit tests in a sub-package of your package (note - this means
that the single Python source file
option above was a trick - you always
need at least one other file for your
unit tests). For example,
Twisted/twisted/test/. Of course, make
it a package with
Twisted/twisted/test/__init__.py.
Place tests in files like
Twisted/twisted/test/test_internet.py.
add Twisted/README and Twisted/setup.py to explain and
install your software, respectively,
if you're feeling nice.
Don't:
put your source in a directory called src or lib. This makes it hard
to run without installing.
put your tests outside of your Python package. This makes it hard to
run the tests against an installed
version.
create a package that only has a __init__.py and then put all your code into __init__.py. Just make a module
instead of a package, it's simpler.
try to come up with magical hacks to make Python able to import your module
or package without having the user add
the directory containing it to their
import path (either via PYTHONPATH or
some other mechanism). You will not
correctly handle all cases and users
will get angry at you when your
software doesn't work in their
environment.
Distutils supports installing modules, packages, and scripts. If you create a distutils setup.py which refers to foo as a package and foo.py as a script, then foo.py should get installed to /usr/local/bin or whatever the appropriate script install path is on the target OS, and the foo package should get installed to the site_packages directory.
You should call the executable just foo, not foo.py, then attempts to import foo will not use it.
As for naming it properly: this is difficult to answer in the abstract; we would need to know what specifically it does. For example, if it configures and controls, calling it -config or ctl might be appropriate. If it is a shell API for the library, it should have the same name as the library.
Your CLI module is one thing, the package that supports it is another thing. Don't confuse the names withe module foo (in a file foo.py) and the package foo (in a directory foo with a file __init__.py).
You have two things named foo: a module and a package. What else do you want to name foo? A class? A function? A variable?
Pick a distinctive name for the foo module or the foo package. foolib, for example, is a popular package name.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
If you're writing a library, or an app, where do the unit test files go?
It's nice to separate the test files from the main app code, but it's awkward to put them into a "tests" subdirectory inside of the app root directory, because it makes it harder to import the modules that you'll be testing.
Is there a best practice here?
For a file module.py, the unit test should normally be called test_module.py, following Pythonic naming conventions.
There are several commonly accepted places to put test_module.py:
In the same directory as module.py.
In ../tests/test_module.py (at the same level as the code directory).
In tests/test_module.py (one level under the code directory).
I prefer #1 for its simplicity of finding the tests and importing them. Whatever build system you're using can easily be configured to run files starting with test_. Actually, the default unittest pattern used for test discovery is test*.py.
Only 1 test file
If there has only 1 test files, putting it in a top-level directory is recommended:
module/
lib/
__init__.py
module.py
test.py
Run the test in CLI
python test.py
Many test files
If has many test files, put it in a tests folder:
module/
lib/
__init__.py
module.py
tests/
test_module.py
test_module_function.py
# test_module.py
import unittest
from lib import module
class TestModule(unittest.TestCase):
def test_module(self):
pass
if __name__ == '__main__':
unittest.main()
Run the test in CLI
# In top-level /module/ folder
python -m tests.test_module
python -m tests.test_module_function
Use unittest discovery
unittest discovery will find all test in package folder.
Create a __init__.py in tests/ folder
module/
lib/
__init__.py
module.py
tests/
__init__.py
test_module.py
test_module_function.py
Run the test in CLI
# In top-level /module/ folder
# -s, --start-directory (default current directory)
# -p, --pattern (default test*.py)
python -m unittest discover
Reference
pytest Good Practices for test layout
unittest
Unit test framework
nose
nose2
pytest
A common practice is to put the tests directory in the same parent directory as your module/package. So if your module was called foo.py your directory layout would look like:
parent_dir/
foo.py
tests/
Of course there is no one way of doing it. You could also make a tests subdirectory and import the module using absolute import.
Wherever you put your tests, I would recommend you use nose to run them. Nose searches through your directories for tests. This way, you can put tests wherever they make the most sense organizationally.
We had the very same question when writing Pythoscope (https://pypi.org/project/pythoscope/), which generates unit tests for Python programs. We polled people on the testing in python list before we chose a directory, there were many different opinions. In the end we chose to put a "tests" directory in the same directory as the source code. In that directory we generate a test file for each module in the parent directory.
I also tend to put my unit tests in the file itself, as Jeremy Cantrell above notes, although I tend to not put the test function in the main body, but rather put everything in an
if __name__ == '__main__':
do tests...
block. This ends up adding documentation to the file as 'example code' for how to use the python file you are testing.
I should add, I tend to write very tight modules/classes. If your modules require very large numbers of tests, you can put them in another, but even then, I'd still add:
if __name__ == '__main__':
import tests.thisModule
tests.thisModule.runtests
This lets anybody reading your source code know where to look for the test code.
Every once in a while I find myself checking out the topic of test placement, and every time the majority recommends a separate folder structure beside the library code, but I find that every time the arguments are the same and are not that convincing. I end up putting my test modules somewhere beside the core modules.
The main reason for doing this is: refactoring.
When I move things around I do want test modules to move with the code; it's easy to lose tests if they are in a separate tree. Let's be honest, sooner or later you end up with a totally different folder structure, like django, flask and many others. Which is fine if you don't care.
The main question you should ask yourself is this:
Am I writing:
a) reusable library or
b) building a project than bundles together some semi-separated modules?
If a:
A separate folder and the extra effort to maintain its structure may be better suited. No one will complain about your tests getting deployed to production.
But it's also just as easy to exclude tests from being distributed when they are mixed with the core folders; put this in the setup.py:
find_packages("src", exclude=["*.tests", "*.tests.*", "tests.*", "tests"])
If b:
You may wish — as every one of us do — that you are writing reusable libraries, but most of the time their life is tied to the life of the project. Ability to easily maintain your project should be a priority.
Then if you did a good job and your module is a good fit for another project, it will probably get copied — not forked or made into a separate library — into this new project, and moving tests that lay beside it in the same folder structure is easy in comparison to fishing up tests in a mess that a separate test folder had become. (You may argue that it shouldn't be a mess in the first place but let's be realistic here).
So the choice is still yours, but I would argue that with mixed up tests you achieve all the same things as with a separate folder, but with less effort on keeping things tidy.
I use a tests/ directory, and then import the main application modules using relative imports. So in MyApp/tests/foo.py, there might be:
from .. import foo
to import the MyApp.foo module.
I don't believe there is an established "best practice".
I put my tests in another directory outside of the app code. I then add the main app directory to sys.path (allowing you to import the modules from anywhere) in my test runner script (which does some other stuff as well) before running all the tests. This way I never have to remove the tests directory from the main code when I release it, saving me time and effort, if an ever so tiny amount.
From my experience in developing Testing frameworks in Python, I would suggest to put python unit tests in a separate directory. Maintain a symmetric directory structure. This would be helpful in packaging just the core libraries and not package the unit tests. Below is implemented through a schematic diagram.
<Main Package>
/ \
/ \
lib tests
/ \
[module1.py, module2.py, [ut_module1.py, ut_module2.py,
module3.py module4.py, ut_module3.py, ut_module.py]
__init__.py]
In this way when you package these libraries using an rpm, you can just package the main library modules (only). This helps maintainability particularly in agile environment.
I recommend you check some main Python projects on GitHub and get some ideas.
When your code gets larger and you add more libraries it's better to create a test folder in the same directory you have setup.py and mirror your project directory structure for each test type (unittest, integration, ...)
For example if you have a directory structure like:
myPackage/
myapp/
moduleA/
__init__.py
module_A.py
moduleB/
__init__.py
module_B.py
setup.py
After adding test folder you will have a directory structure like:
myPackage/
myapp/
moduleA/
__init__.py
module_A.py
moduleB/
__init__.py
module_B.py
test/
unit/
myapp/
moduleA/
module_A_test.py
moduleB/
module_B_test.py
integration/
myapp/
moduleA/
module_A_test.py
moduleB/
module_B_test.py
setup.py
Many properly written Python packages uses the same structure. A very good example is the Boto package.
Check https://github.com/boto/boto
How I do it...
Folder structure:
project/
src/
code.py
tests/
setup.py
Setup.py points to src/ as the location containing my projects modules, then i run:
setup.py develop
Which adds my project into site-packages, pointing to my working copy. To run my tests i use:
setup.py tests
Using whichever test runner I've configured.
I prefer toplevel tests directory. This does mean imports become a little more difficult. For that I have two solutions:
Use setuptools. Then you can pass test_suite='tests.runalltests.suite' into setup(), and can run the tests simply: python setup.py test
Set PYTHONPATH when running the tests: PYTHONPATH=. python tests/runalltests.py
Here's how that stuff is supported by code in M2Crypto:
http://svn.osafoundation.org/m2crypto/trunk/setup.py
http://svn.osafoundation.org/m2crypto/trunk/tests/alltests.py
If you prefer to run tests with nosetests you might need do something a little different.
I put my tests in the same directory as the code under test (CUT). In projects where I can tweak pytest with my plugin, for foo.py I use foo.pt for the tests which makes editing a particular module and its test together really easy: vi foo.*.
Where I can't do this, I use foo_ut.py or similar. You can still use vi foo* though that will also catch foobar.py and foobar_ut.py if those exist.
In either case I tweak the test discovery process to find these.
This puts the tests right beside the code in a directory listing, making it obvious that tests are there, and makes opening the tests as easy as it can possibly be when they're in a separate file. (For editors started from the command line, as described above; for GUI systems, click on the code file and the adjacent (or very nearly adjacent) test file.
As others have pointed out, this also makes it easier to refactor and to extract the code for use elsewhere should that ever be necessary.
I really dislike the idea of putting tests in a completely different directory tree; why make it harder than necessary for developers to open up the tests when they're opening the file with the CUT? It's not like the vast majority of developers are so keen on writing or tweaking tests that they'll ignore any barrier to doing that, instead of using the barrier as an excuse. (Quite the opposite, in my experience; even when you make it as easy as possible I know many developers who can't be bothered to write tests.)
We use
app/src/code.py
app/testing/code_test.py
app/docs/..
In each test file we insert ../src/ in sys.path. It's not the nicest solution but works. I think it would be great if someone came up w/ something like maven in java that gives you standard conventions that just work, no matter what project you work on.
If the tests are simple, simply put them in the docstring -- most of the test frameworks for Python will be able to use that:
>>> import module
>>> module.method('test')
'testresult'
For other more involved tests, I'd put them either in ../tests/test_module.py or in tests/test_module.py.
In C#, I've generally separated the tests into a separate assembly.
In Python -- so far -- I've tended to either write doctests, where the test is in the docstring of a function, or put them in the if __name__ == "__main__" block at the bottom of the module.
When writing a package called "foo", I will put unit tests into a separate package "foo_test". Modules and subpackages will then have the same name as the SUT package module. E.g. tests for a module foo.x.y are found in foo_test.x.y. The __init__.py files of each testing package then contain an AllTests suite that includes all test suites of the package. setuptools provides a convenient way to specify the main testing package, so that after "python setup.py develop" you can just use "python setup.py test" or "python setup.py test -s foo_test.x.SomeTestSuite" to the just a specific suite.
I've recently started to program in Python, so I've not really had chance to find out best practice yet.
But, I've written a module that goes and finds all the tests and runs them.
So, I have:
app/
appfile.py
test/
appfileTest.py
I'll have to see how it goes as I progress to larger projects.