I have a directory structure like this:
|-root
|-app
program.py
tests.txt
|-tests
runTests.py
My tests.txt file contains all the doctests on my program.py code. It calls
from program import *
and then it makes all the doctest calls.
My runTests.py file has this code:
import doctest
doctest.testfile("app/tests.txt")
In the command line I then call:
python runTests.py
and it does indeed find the tests.txt file and reads it successfully but it does not find the module "program" which I am trying to import. What am I doing wrong? How can I have them in separate directories and still be able to run the tests?
Thanks
There are two ways to do this:
You can add it the sys.path in runTests.py: import sys; sys.path.append('app')
The same can be achieved with the environment variable PYTHONPATH.
You can turn the folder into a package by adding a file __init__.py and importing from app.program import * in tests.txt.
Reminder: The main premise of doctest is that the method/function documentation (what you get when you look at the __doc__ property) explains what it does and gives examples (the tests).
Doctest then finds such code and executes it, making sure that the examples in the documentation are actually working.
If you move the tests out, then you're taking away a major source of information from the user of your code. I guess it might make sense if you have extensive documentation outside of the source code or if you have many additional tests (you want to give users a few examples, not all 500 unit tests which drive code coverage to 100%).
That said, to fix the issue, you need to make the import work. The folder app doesn't magically appear in the search path for modules. You have to tell Python that this is in fact a place where it should look.
Related
I'm currently trying to import one of my modules from a different folder.
Like this...
from Assets.resources.libs.pout import Printer, ForeColor, BackColor
This import method works completely fine in PyCharm, however, when i try to launch the file in cmd or IDLE, i get this error.
ModuleNotFoundError: No module named 'Assets'
This is my file structure from main.py to pout.py:
- Assets
- main.py
- resources
- libs
- pout.py
Any clue about how i could fix this ?
Any help is appreciated !
Edit: The original answer was based on the assumption that the script you're running is within the folder structure given, which a re-read tells me may not be true. The general solution is to do
sys.path.append('path_to_Assets')
but read below for more detail.
Original answer
The paths that modules can be loaded from will differ between the two methods of running the script.
If you add
import sys
print(sys.path)
to the top of your script before the imports you should be able to see the difference.
When you run it yourself the first entry will be the location of the script itself, followed by various system/environment paths. When you run it in PyCharm you will see the same first entry, followed by an entry for the top level of the project. This is how it finds the modules when run from PyCharm. This behaviour is controlled by the "Add content roots to PYTHONPATH" option in the run configuration.
Adding this path programmatically in a way that will work in all situations isn't trivial because, unlike PyCharm, your script doesn't have a concept of where the top level should be. BUT if you know you'll be running the script from the top level, i.e. your working directory will be the folder containing Assets and you're running something like python Assets/main.py then you can do
sys.path.append(os.path.abspath('.'))
and that will add the correct folder to the path.
Appending sys path didn't work for me on windows, hence here is the solution that worked for me:
Add an empty __init__.py file to each directory
i.e. in Assets, resources, libs.
Then try importing with only the base package names.
Worked for me!
I have the following folder structure:
app
__init__.py
utils
__init__.py
transform.py
products
__init__.py
fish.py
In fish.py I'm importing transform as following: import utils.transform.
When I'm running fish.py from Pycharm, it works perfectly fine. However when I am running fish.py from the Terminal, I am getting error ModuleNotFoundError: No module named 'utils'.
Command I use in Terminal: from app folder python products/fish.py.
I've already looked into the solutions suggested here: Importing files from different folder, adding a path to the application folder into the sys.path helps. However I am wondering if there is any other way of making it work without adding two lines of code into the fish.py. It's because I have many scripts in the /products directory, and do not want to add 2 lines of code into each of them.
I looked into some open source projects, and I saw many examples of importing modules from a parallel folder without adding anything into sys.path, e.g. here:
https://github.com/jakubroztocil/httpie/blob/master/httpie/plugins/builtin.py#L5
How to make it work for my project in the same way?
You probably want to run python -m products.fish. The difference between that and python products/fish.py is that the former is roughly equivalent to doing import products.fish in the shell (but with __name__ set to __main__), while the latter does not have awareness of its place in a package hierarchy.
This expands on #Mad Physicist's answer.
First, assuming app is itself a package (since you added __init__.py to it) and utils and products are its subpackages, you should change the import to import app.utils.transform, and run Python from the root directory (the parent of app). The rest of this answer assumes you've done this. (If it wasn't your intention making app the root package, tell me in a comment.)
The problem is that you're running app.products.fish as if it were a script, i.e. by giving the full path of the file to the python command:
python app/products/fish.py
This makes Python think this fish.py file is a standalone script that isn't part of any package. As defined in the docs (see here, under <script>), this means that Python will search for modules in the same directory as the script, i.e. app/products/:
If the script name refers directly to a Python file, the directory
containing that file is added to the start of sys.path, and the file
is executed as the __main__ module.
But of course, the app folder is not in app/products/, so it will throw an error if you try to import app or any subpackage (e.g. app.utils).
The correct way to start a script that is part of a package is to use the -m (module) switch (reference), which takes a module path as an argument and executes that module as a script (but keeping the current working directory as a module search path):
If this option is given, [...] the current directory
will be added to the start of sys.path.
So you should use the following to start your program:
python -m app.products.fish
Now when app.products.fish tries to import the app.utils.transform module, it will search for app in your current working directory (which contains the app/... tree) and succeed.
As a personal recommendation: don't put runnable scripts inside packages. Use packages only to store all the logic and functionality (functions, classes, constants, etc.) and write a separate script to run your application as you wish, putting it outside the package. This will save you from this kind of problems (including the double import trap), and has also the advantage that you can write several run configurations for the same package by just making a separate startup script for each.
I'm trying to run a python file, but I keep getting a ImportError.
My set up is I have a project with the following path:
/Users/John/Documents/pythonprojects/projectX
within 'projectX' I have a folder called 'components' which contains two python files titled 'py_file' and 'init'.
/Users/John/Documents/pythonprojects/projectX/components
At the top of 'py_file', I'm importing a namedTuple that is defined in 'init'
`from components import some_tuple`
When I run python py_file.py run I get ImportError: No module named components
However, if I add the lines below, I can get the file to run:
import sys
sys.path.append("..")
Any idea whats going on with this?
You should probably make sure your PYTHONPATH environment variable is set to /Users/John/Documents/pythonprojects/projectX.
Effectively, that is what you do with that line sys.path.append(".."), since the relative .. directory from py_file.py is /Users/John/Documents/pythonprojects/projectX.
from components import some_tuple
Python is looking for components in the directory you started it from. But, you started Python in your components directory, so that's not going to work. Appending ".." to your path appears to fix the problem because now Python will look in the parent directory, which does contain components.
However, the better approach is to leave sys.path alone. Figure out which .py file in your program is going to be the entry point, and make all your imports work when it executes from there. Then, always execute your program from there. If you want to quickly test some small module off in a package somewhere, write some tests!
If you'd like the flexibility of beginning execution from anywhere, you can look at relative imports. Personally I think these are better left until after absolute imports make sense.
I've worked on several medium-sized python applications to date, and every time it seems like I cobble together a terrible system of imports from tangential Stack Overflow answers and half-understood blog posts. It's ugly and hard to maintain and ultimately very unsatisfying. With this question I attempt to put all that behind me.
Say I have a python application split into the following files:
app.py
constants.py
ui/window.py
web/connection.py
With the following include requirements:
app.py needs to include window.py and connection.py
window.py needs to include constants.py and connection.py
connection.py needs to include constants.py
app.py is the starting point for the application, but window.py and connection.py are also invokable from the command line to test basic functionality (ideally from within their respective folders).
What combination of __init__.py files, carefully crafted import statements and wacky python path magic will allow me to achieve this structure?
Thanks very much,
--Dan
It really helps if, instead of thinking in terms of "file structure" first and then trying to figure out the packages, you design things in terms of packages, and then lay out your file structure to implement those packages.
But if you want to know how to hack up what you already have: If you put this at the top level (that is, in one of the paths on your sys.path), and create files names ui/__init__.py and web/__init__.py, then:
app.py can be run as a script.
app.py can be run with -m app.
app.py can be imported with import app.
window.py cannot be run directly.
window.py can be run with -m ui.window.
window.py can be imported with import ui.window.
connection.py cannot be run directly.
connection.py can be run with -m web.connection.
connection.py can be imported with import web.connection.
No wacky path magic is needed; you just need the top level (with app.py, constants.py, ui, and web) to be on your sys.path—which it automatically is when you run with that directory as your working directory, or install everything directly into site-packages, or install it as an egg, etc.
That's as close as you're going to get to what you want. You do ever want to run code with a package directory as your current working directory or otherwise on sys.path, so don't even try. If you think you need that, what you probably want is to separate the runnable code out into a script that you can put at the top level, or somewhere entirely separate. (For example, look at pip or ipython, which installs scripts into somewhere on your system $PATH that do nothing but import some module and run a function.)
The only other thing you might want to consider is putting all of this into a package, say, myapp. You do that by adding a top-level __init__.py, and then running from the parent directory, and adding myapp. to the start of all your importand -m statements. That means you can no longer run app.py as a script either, so again you will need to split the script code out into a separate file from the module that does all the work.
You can use that structure with just a small modification: add empty __init__.py files to the ui/ and web/ directory. Then, where you would have done import window, do either import ui.window, or from ui import window. Similarly, change import connection to import web.connection or from web import connection.
Rationale: Python doesn't work so much with directories as it does with packages, which are directories with an __init__.py in them. By changing ui and web to be packages, you don't have to do any particular Python path magic to work with them, and you get the benefit of adding some structure to your modules and imports. That will become particularly important if you start having modules with the same name in different directories (e.g. a util.py in both the ui and web directories; not necessarily the cleanest design but you get the idea).
If you invoke window.py or connection.py directly to test them, you need to add the top-level directory to your PYTHONPATH for things to still work – but there is a subtle additional wrinkle. When you run this from the top-level directory:
PYTHONPATH=$PWD python web/connection.py
you now have both the top-level directory on your module path AND the web/ directory. This can cause certain relative imports to do unexpected things.
Another way is to use Python's -m option from the top-level directory:
python -m web.foo
I know many folks like to nail their tests right into the modules like this, but I should also note that there are other ways to structure your tests, particularly with an updated unittest library and tools like nosetests, that will make it a little bit easier to run your tests as your project gets larger. See the skeleton here for a reasonable example:
http://learnpythonthehardway.org/book/ex46.html
How do I run trial so that it executes all tests within a directory? All my unit tests pass if I run trial on each file individually, but if I try something like...
trial test/
on the test directory, it gives me one "PASSED", and the following message...
UserWarning: (for module __init__) not in path importer cache (PEP 302 violation
- check your local configuration).
rather than actually running all the tests within the directory.
First of all: you can't call your top-level unit test package test. That's the name of Python's unit tests, so you will never be able to run your tests in an installed configuration, and depending on how your python is set up, you may end up importing python's own tests instead of your own.
Second: sys.path is a vast and subtle mystery.
trial supports running on files and directories as a quick getting-started hack, but it can never really be completely correct about using path names. The right thing to do is to pass trial a module (or package) name, that it can import as a python module and inspect.
So if your directory structure looks like:
~/Projects/MyProject/
~/Projects/MyProject/myproject/
~/Projects/MyProject/myproject/__init__.py
~/Projects/MyProject/myproject/stuff.py
~/Projects/MyProject/myproject/test/
~/Projects/MyProject/myproject/test/__init__.py
~/Projects/MyProject/myproject/test/test_stuff.py
then you should run your tests like this:
PYTHONPATH=$HOME/Projects/MyProject (cd /tmp; trial myproject.test)
in other words, don't run your tests from within your project's directory; this dumps _trial_temp directories all over your source code, confuses "the place I load my code from" and "the current directory" and generally makes a muddle of various things which can be difficult to untangle later.
So, set up your PYTHONPATH and PATH using the path-management tool of your choice: Combinator, setup.py develop, virtualenv – or just dumping junk into your ~/.bashrc – and then run trial from some temporary location, on a uniquely-named top-level Python package, and everything should work just fine.