We Keep Coding

python import relative path 'No module found error'

I am in folder2 trying to execute trigger.py
I used:
from ..folder1.subfolder1.function1 import *
from ..folder1.test.py
got error attempted relative import with no known parent package

If you're running out of folder2, then for all practical purposes, trigger.py is a module on its own, not part of a package hierarchy, and can't use relative imports to folder1 (because they're unrelated).
If you can, make your working directory the parent directory of Project, and invoke your script with:
python3 -m Project.folder2.trigger
which runs a module as the main entry point via import syntax, without losing the information on the package structure.
If the whole project is actually installed to system or user site-packages, your working directory won't matter (it's only needed if it's not in sys.path automatically, because the working directory itself is included in sys.path), and the invocation will work anywhere.

ModuleNotFoundError when running script from Terminal

I have the following folder structure:
app
__init__.py
utils
__init__.py
transform.py
products
__init__.py
fish.py
In fish.py I'm importing transform as following: import utils.transform.
When I'm running fish.py from Pycharm, it works perfectly fine. However when I am running fish.py from the Terminal, I am getting error ModuleNotFoundError: No module named 'utils'.
Command I use in Terminal: from app folder python products/fish.py.
I've already looked into the solutions suggested here: Importing files from different folder, adding a path to the application folder into the sys.path helps. However I am wondering if there is any other way of making it work without adding two lines of code into the fish.py. It's because I have many scripts in the /products directory, and do not want to add 2 lines of code into each of them.
I looked into some open source projects, and I saw many examples of importing modules from a parallel folder without adding anything into sys.path, e.g. here:
https://github.com/jakubroztocil/httpie/blob/master/httpie/plugins/builtin.py#L5
How to make it work for my project in the same way?

You probably want to run python -m products.fish. The difference between that and python products/fish.py is that the former is roughly equivalent to doing import products.fish in the shell (but with __name__ set to __main__), while the latter does not have awareness of its place in a package hierarchy.

This expands on #Mad Physicist's answer.
First, assuming app is itself a package (since you added __init__.py to it) and utils and products are its subpackages, you should change the import to import app.utils.transform, and run Python from the root directory (the parent of app). The rest of this answer assumes you've done this. (If it wasn't your intention making app the root package, tell me in a comment.)
The problem is that you're running app.products.fish as if it were a script, i.e. by giving the full path of the file to the python command:
python app/products/fish.py
This makes Python think this fish.py file is a standalone script that isn't part of any package. As defined in the docs (see here, under <script>), this means that Python will search for modules in the same directory as the script, i.e. app/products/:
If the script name refers directly to a Python file, the directory
containing that file is added to the start of sys.path, and the file
is executed as the __main__ module.
But of course, the app folder is not in app/products/, so it will throw an error if you try to import app or any subpackage (e.g. app.utils).
The correct way to start a script that is part of a package is to use the -m (module) switch (reference), which takes a module path as an argument and executes that module as a script (but keeping the current working directory as a module search path):
If this option is given, [...] the current directory
will be added to the start of sys.path.
So you should use the following to start your program:
python -m app.products.fish
Now when app.products.fish tries to import the app.utils.transform module, it will search for app in your current working directory (which contains the app/... tree) and succeed.
As a personal recommendation: don't put runnable scripts inside packages. Use packages only to store all the logic and functionality (functions, classes, constants, etc.) and write a separate script to run your application as you wish, putting it outside the package. This will save you from this kind of problems (including the double import trap), and has also the advantage that you can write several run configurations for the same package by just making a separate startup script for each.

IntelliJ Python unittests cannot import modules

I'm trying to configure my IntelliJ Ultimate 2017 IDE to run Python unit tests. It's not working.
The project is a Gradle project, and most modules are Java/Scala. There are several Python modules though.
The Python plugin is installed and the Python facet added to the appropriate modules.
The Python Interpreter is set up fine. The IDE can actually move around in the code with a ctrl-click action.
Under 'Project Structure' -> '<mymodule>' -> dependencies both the interpreter and the <Module source> are included.
Each Python module has a 'content root'. Immediately under this is a src folder, under which there is a test folder. The contents of the test folder mirror the src folder, except that files are prepended with test_ to avoid Python name collisions.
The test configuration has both 'Add source/content roots to PYTHONPATH' options checked. The interpreter is specified.
If test is marked as a tests folder within the module settings, I can right click and 'run tests' for the tests in that folder. However, ImportErrors occur, because it can't find modules within the src directory.
There are __init__.py files in all the appropriate places.
IntelliJ gives you what it is running, and if I run that on the command, it works, something like:
python -m unittest discover -s /Users/nathanielford/myrepo/mymodule/src/tests -p test_util.py
It appears as though the root path that is being used is set to wherever the unittest file I'm running is: it doesn't use src as the path, and so it cannot find things to import. There is no obvious place to specify this behavior. It didn't seem to work like this in the past.
This started happening recently, after an IntelliJ update. It's possible some configuration became screwy and needs to be fixed, but I don't know what that is. It's also possible that some bug was introduced into IntelliJ, but I don't know how to identify that, either. What am I missing?

Had the same issue and resolved it by no marking the test folder as excluded but just moving it out of the source.
So if the structure is:
src
--->package
--->resources
test
mark in IntelliJ src folder as Sources and src/resources as Resources.
You tests will run from within IDEA.
As I am new to python, I am not sure whether this will affect the packaging. Doing a build though doesn't export the test folder.
Hope that helps.

I have similar problems: cannot import manually installed module.
I try to add classpath in Mac environment parameters and intellij SDKs, but it doesn't work.
My final solution is to add the classpath of the module to Run/Debug Configurations:
Open Run/Debug Configurations and select your unittest class:
Run --> Edit Configurations...
Add your module's classpath to Environment variables:

Handling complicated directory structure with python imports

I've worked on several medium-sized python applications to date, and every time it seems like I cobble together a terrible system of imports from tangential Stack Overflow answers and half-understood blog posts. It's ugly and hard to maintain and ultimately very unsatisfying. With this question I attempt to put all that behind me.
Say I have a python application split into the following files:
app.py
constants.py
ui/window.py
web/connection.py
With the following include requirements:
app.py needs to include window.py and connection.py
window.py needs to include constants.py and connection.py
connection.py needs to include constants.py
app.py is the starting point for the application, but window.py and connection.py are also invokable from the command line to test basic functionality (ideally from within their respective folders).
What combination of __init__.py files, carefully crafted import statements and wacky python path magic will allow me to achieve this structure?
Thanks very much,
--Dan

It really helps if, instead of thinking in terms of "file structure" first and then trying to figure out the packages, you design things in terms of packages, and then lay out your file structure to implement those packages.
But if you want to know how to hack up what you already have: If you put this at the top level (that is, in one of the paths on your sys.path), and create files names ui/__init__.py and web/__init__.py, then:
app.py can be run as a script.
app.py can be run with -m app.
app.py can be imported with import app.
window.py cannot be run directly.
window.py can be run with -m ui.window.
window.py can be imported with import ui.window.
connection.py cannot be run directly.
connection.py can be run with -m web.connection.
connection.py can be imported with import web.connection.
No wacky path magic is needed; you just need the top level (with app.py, constants.py, ui, and web) to be on your sys.path—which it automatically is when you run with that directory as your working directory, or install everything directly into site-packages, or install it as an egg, etc.
That's as close as you're going to get to what you want. You do ever want to run code with a package directory as your current working directory or otherwise on sys.path, so don't even try. If you think you need that, what you probably want is to separate the runnable code out into a script that you can put at the top level, or somewhere entirely separate. (For example, look at pip or ipython, which installs scripts into somewhere on your system $PATH that do nothing but import some module and run a function.)
The only other thing you might want to consider is putting all of this into a package, say, myapp. You do that by adding a top-level __init__.py, and then running from the parent directory, and adding myapp. to the start of all your importand -m statements. That means you can no longer run app.py as a script either, so again you will need to split the script code out into a separate file from the module that does all the work.

You can use that structure with just a small modification: add empty __init__.py files to the ui/ and web/ directory. Then, where you would have done import window, do either import ui.window, or from ui import window. Similarly, change import connection to import web.connection or from web import connection.
Rationale: Python doesn't work so much with directories as it does with packages, which are directories with an __init__.py in them. By changing ui and web to be packages, you don't have to do any particular Python path magic to work with them, and you get the benefit of adding some structure to your modules and imports. That will become particularly important if you start having modules with the same name in different directories (e.g. a util.py in both the ui and web directories; not necessarily the cleanest design but you get the idea).
If you invoke window.py or connection.py directly to test them, you need to add the top-level directory to your PYTHONPATH for things to still work – but there is a subtle additional wrinkle. When you run this from the top-level directory:
PYTHONPATH=$PWD python web/connection.py
you now have both the top-level directory on your module path AND the web/ directory. This can cause certain relative imports to do unexpected things.
Another way is to use Python's -m option from the top-level directory:
python -m web.foo
I know many folks like to nail their tests right into the modules like this, but I should also note that there are other ways to structure your tests, particularly with an updated unittest library and tools like nosetests, that will make it a little bit easier to run your tests as your project gets larger. See the skeleton here for a reasonable example:
http://learnpythonthehardway.org/book/ex46.html

How do I run twisted trial on all tests in a directory?

How do I run trial so that it executes all tests within a directory? All my unit tests pass if I run trial on each file individually, but if I try something like...
trial test/
on the test directory, it gives me one "PASSED", and the following message...
UserWarning: (for module __init__) not in path importer cache (PEP 302 violation
- check your local configuration).
rather than actually running all the tests within the directory.

First of all: you can't call your top-level unit test package test. That's the name of Python's unit tests, so you will never be able to run your tests in an installed configuration, and depending on how your python is set up, you may end up importing python's own tests instead of your own.
Second: sys.path is a vast and subtle mystery.
trial supports running on files and directories as a quick getting-started hack, but it can never really be completely correct about using path names. The right thing to do is to pass trial a module (or package) name, that it can import as a python module and inspect.
So if your directory structure looks like:
~/Projects/MyProject/
~/Projects/MyProject/myproject/
~/Projects/MyProject/myproject/__init__.py
~/Projects/MyProject/myproject/stuff.py
~/Projects/MyProject/myproject/test/
~/Projects/MyProject/myproject/test/__init__.py
~/Projects/MyProject/myproject/test/test_stuff.py
then you should run your tests like this:
PYTHONPATH=$HOME/Projects/MyProject (cd /tmp; trial myproject.test)
in other words, don't run your tests from within your project's directory; this dumps _trial_temp directories all over your source code, confuses "the place I load my code from" and "the current directory" and generally makes a muddle of various things which can be difficult to untangle later.
So, set up your PYTHONPATH and PATH using the path-management tool of your choice: Combinator, setup.py develop, virtualenv – or just dumping junk into your ~/.bashrc – and then run trial from some temporary location, on a uniquely-named top-level Python package, and everything should work just fine.