We Keep Coding

Python imports work in GCP/local container, but not with directly calling the scripts

I have built an API that is a HTTP function working in GCP, dev deployment works well, local Docker containers with it work also fine.
When I try to call one of the particular libraries individually, especially since I started writing unit tests now, I get multiple import errors.
My folder structure is
├── src/
│ ├──__init__.py
│ ├──functions.py
│ ├──ml.py
│
├──connections/
│ ├──__init__.py
│ ├──connect.py
│ ├──external_calls.py
│
│
├──tests/
│ ├──__init__.py
│ ├──tests1.py
│
├── __init__.py
├── main.py
├── conf.py
(+ other irrelevant files in main for deployment or requirements)
functions.py has imports
from .external_calls import x,y,z
from connections.connect import x,y,z
main also has imports such
from src.functions import x,y
This works well in the cloud or docker container. When I try to run functions separately or via tests1.py, it does not allow me to import them!
In tests1.py I am aiming to run functions from functions.py for unit tests.
There, when I try 'from src.functions import x,y,z' I get 'no module names functions'.
When I do 'from .src.functions import x,y,z' I get 'atempted relative imports with no known parent package'.
I tried to run it from regular python, as package with -m flag and using pytest. They all failed.
Empty init.py files are in each folder
Finally, I did override paths in tests1.py, which is a bad practice afaik but could not work out anything else. I used sys.path.append("../src") and tried append("src"). This only fixed importing functions.py. From there, I get error when functions.py tries to import .external_calls and connections.connect... I do not want to override paths in non-test scripts, as this will work out very bad, and they do work in dev.
I have tried everything that I could find and feel a bit defeated, would appreciate any help !

python import module - what am I missing?

I tried to restructure my project for my own Tetris.
With changing it to different files and also different folders I got confused about the module imports.
This is the directory
Jstris
├── code_base
│ ├── constants.py
│ ├── functions.py
│ └── tetromino.py
│
├── game_mode
│ ├── free_play.py
│ └── sprint.py
└── run.py
The main goal is to execute the run.py file and it works as it should. Just to be clear, the whole game is working. I get to a menu and can start all my different games. But I want to understand what is happening here and can't figure it out myself.
In run.py I import e.g.: (not the original code, just for demonstration)
from code_base.constants import SCREEN_HEIGHT, SCREEN_WIDTH
from code_base.functions import draw_window
from game_mode.free_play import main_free_play
In functions.py I import the following:
from code_base.constants import PLAY_WIDTH, PLAY_HEIGHT
from code_base.tetromino import Tetromino
I need to have the syntax like above in functions.py to not get an error when executing run.py.
But if I want to execute functions.py I get an error "ModuleNotFoundError: No module named 'code_base'.
Do I have to accept it because I won't ever execute functions.py or is there a way my whole projects works but still every module itself is able to work fine when executing ?
Or one step further, is my way to structure the files and the importing just circuitous/ not Python convention? I'm using python 3.8

this is because of relative path, when you run your "run.py" file, it understand code_base.functions because it sees 2 modules (code_base and game_mode), which are your folder names in Jstris folder, but when you import the same name in functions.py it looks for a code_base folder and doesn't find one, so it raises an error
if you want all your modules to run by themselves, you can try absolute paths, which is covered in this issue : How to import a module given the full path?

Is runpy safe to use?

I am writing some code that will require data from downstream the package hierarchy. Consider the following project structure:
├── app/
│ ├── main.py
│ ├── lib.py
│ └── subpck/
│ └── module.py
And the following code:
# lib.py
some_instance = SomeClass()
# subpck.module
from ..lib import some_instance
some_instance.a = 'abc'
# main.py
from .lib import some_instance
print(some_instance.a)
The key point here is that main.py does not import subpck.module, so the latter's code is not run. However, I have successfully used runpy in main.py to run subpck.module and have gotten the desired results.
My question is this:
If I can somehow figure out how to encapsulate the use of runpy in SomeClass, is it safe to do?
I have been reading the runpy docs, but am nervous if I am missing something. I also haven't heard if this is a big "no-no" for code that may be used in production.
Any help is appreciated.

Failed to import python module from different directory

I have this code structure in python3:
- datalake
__init__.py
utils
__init__.py
utils.py
lambdas
__init__.py
my-lambdas.py
- tests
__init__.py
demo.py
All init__.py files are empty.
My problem is how I can import datalake module from tests/demo.py?
I tried from datalake.utils import utils in demo.py but when I run python tests/demo.py from command line, I get this error ModuleNotFoundError: No module named 'datalake'.
If I use this code:
from ..datalake.utils import utils
I will get error ValueError: attempted relative import beyond top-level package.
I also tried to import the module utils from my-lambda.py file which also failed. The code in my-lambda.py is from datalake.utils import utils but I get ModuleNotFoundError: No module named 'datalake' error when run python datalake/lambda/my-lambda.py from command line.
How can I import the module?

When you run a command like python tests/demo.py, the folder you are in does not get added to the PYTHONPATH, the script folder does. So a top-level import like import datalake will fail. To get around this you can run your tests as a module:
Python 2:
python -m tests/demo
Python 3:
python -m tests.demo
and any datalake imports in demo.py will work.
It sounds like what you really want to do is have a folder with tests separate to your main application and run them. For this I recommend py.test, for your case you can read Tests Outside Application Code for how to do it. TL;DR is run your tests from your top level project folder with python -m py.test and it will work.

First of all, my-lambdas.py is not importable with the import statement as hyphens are not valid in Python identifiers. Try to follow PEP-8's naming conventions, such as mylambdas.py.
Otherwise the package structure looks good, and it should be importable as long as you are at the level above datalake/, e.g., if you were in the directory myproject/ below:
myproject
├── datalake
│ ├── __init__.py
│ ├── utils
│ │ ├── __init__.py
│ │ └── utils.py
│ └── lambdas
│ ├── __init__.py
│ └── mylambdas.py
└── tests
├── __init__.py
└── demo.py
Then this should work:
~/myproject$ python -c 'from datalake import utils'
Otherwise, setting the environment variable PYTHONPATH to the path above datalake/ or modifying sys.path are both ways of changing where Python can import from. See the official tutorial on modules for more information.
Also some general advice: I've found it useful to stick with simple modules rather than packages (directories) until there is a need to expand. Then you can change foo.py into a foo/ directory with an __init__.py file and import foo will work as before, although you may need to add some imports to the __init__.py to maintain API compatibility. This would leave you with a simpler structure:
myproject
├── datalake
│ ├── __init__.py
│ ├── utils.py
│ └── lambdas.py
└── tests
├── __init__.py
└── demo.py

You can add the module directory into your sys.path:
import sys
sys.path.append("your/own/modules/folder") # like sys.path.append("../tests")
but this is a one-shot method, which is just valid at this time, the added path is not permanent, it will be eliminated after the code completed execution.

One of the ways to import the file directly instead of using from, like import util

you can try run :
python -m datalake.lambda.my-lambda
follow: https://docs.python.org/3.7/using/cmdline.html#cmdoption-m

Execute module as script breaks imports

I'm trying to build a Python program with the "structure" shown at the end.
The problem is that actions should be able to be executed as scripts as well (I've already included a main in them). When I try to execute DummyAction.py imports keep complaining they can't find misc.
How can I use DummyAction.py as a script and still use the functions in utils.py?
DummyAction.py contains a class called DummyActionClass, and the same for DummyTrigger.py. In utils.py there are several functions that both actions and triggers use and MMD.py contains the main.
/MMD
├── __init__.py
├── MMD.py
├── /actions
│   ├── __init__.py
│   ├── DummyAction.py
├── /misc
│   ├── __init__.py
│   ├── utils.py
└── /triggers
├── DummyTrigger.py
└── __init__.py
The import in DummyAction.py and DummyTrigger.py is:
from misc import utils
And the error is:
File "DDM/actions/DummyAction.py", line 11, in <module>
from misc import utils
ImportError: No module named misc

Seen the updated question, I think the problem is that you should do the import including the root of your dependecies tree: MMD.
So they should all look like:
from MMD.misc import utils
And also you need to call python with the -m option:
python -m MMD.actions.DummyAction
Edit: You said that MMD.py contains the main but it can't be your executable, and that's because is a module (is inside a directory with an __init__.py file). MMD is like your library so you need the executable to be outside and use such library.
You can find [here] some guidelines on how to organize your project.
If you can change you project structure I'll suggest to do it like this:
MMD/
├── runner.py
└── mmd
├── __init__.py
├── main.py
├── /actions
│ ├── __init__.py
│ ├── DummyAction.py
├── /misc
│ ├── __init__.py
│ ├── utils.py
└── /triggers
├── DummyTrigger.py
└── __init__.py
Then in any file inside the mmd directory every import should start with mmd, for example:
from mmd.misc import utils
from mmd.actions import DummyActions
And you put your main code that now is inside MMD.py inside a Main class in main.py, with something like:
# main.py
from mmd.misc import utils
class Main:
def start_session(self):
utils.function()
# etc ...
And then in runner.py you do something like:
# runner.py
from mmd.main import Main
cli = Main()
cli.start_session()
This way inside the MMD directory calling python runner.py you would execute your code, and you can also make executable runner.py so a simply ./runner.py will run your code.
And run your module with:
python -m mmd.actions.DummyAction
I'd do it like this, becasue this way is open to future implementation (and is almost like in the line guides).
If instead you can't, then you can give it a try at removing __init__.py from the MMD directory.

I assume this is a directory structure you're talking about, in which case python doesn't know where to look for utils.py - it tries the local directory, a few places in the path then gives up. It's simple enough to modify the path:
import sys
sys.path.append("/MMD/misc")
import utils
and you should be away.

I've found a "workarround"
try:
#When executing from the main
from misc import utils
except:
#When executing as a standalone script
from MMD.misc import utils
that allows to:
Call the module as a script (while using other modules): python -m MMD.actions.DummyAction
Call the main program and use the module: python MMD/MMD.py
Although I'm not sure if it's strictly correct to use a try - except block with an import, so feel free to add comments or other solutions.
The complete solution would be:
MMD.main
from misc import utils
from actions import DummyAction
class MMD():
def __init__(self):
a = DummyAction.DummyActionClass()
utils.foo()
if __name__ == '__main__':
d = MMD()
Then in actions/DummyAction.py:
try:
#When executing from the main
from misc import utils
except:
#When executing as a standalone script
from MMD.misc import utils
class DummyActionClass():
def __init__(self):
utils.foo()
if __name__ == '__main__':
a = DummyActionClass()
And finally in misc/utils.py:
def foo():
print "Foo was called"