I am having some trouble with local modules when deploying on Cloud Functions - any ideas or best practices would be appreciated!
I am trying to deploy a piece of my project as a Cloud Function. It uses some local code from the project, which is shared with other modules - and I use an absolute import for that. I am using a Cloud Repository for deployment, and there I state the folder where the function resides (parent\cloud_function\). The problem is the parent package is not available with that setup.
This is an example of the project structure:
├── parent_repo
│ ├── parent
│ │ ├── __init__.py
│ │ ├── config.conf
│ │ ├── config.py
│ │ ├── cloud_function
│ │ │ ├── __init__.py
│ │ │ ├── main.py
│ │ │ └── requirements.txt
│ │ ├── shared_module
│ │ │ ├── __init__.py
│ │ │ ├── package1.py
│ │ │ └── package2.py
│ │ ├── other_module
│ │ │ ├── __init__.py
│ │ │ ├── some_script.py
│ │ │ └── another_script.py
│ │ └── utils.py
inside parent.cloud_function.main.py AND in parent.other_module.some_script.py I use:
from parent.shared_module.package1 import some_func
from parent.shared_module.package2 importsome_class
to access shared code. However, when trying to deploy the function on Cloud Functions, since I assume it only looks at the folder alone, the parent module is unavailable.
Of course I could simply nest all required code inside the cloud_function folder - but from a project perspective that isn't ideal - as that code is shared across other resources, and does not logically belong there.
Does anyone have a good idea how to this better?
Thanks in advance!
Very shortly - it is difficult.
Here is Python runtime - Specifying dependencies description -
requirements.txt file or packaging local dependencies alongside your
function
You probably can play with Private dependencies in the Cloud Build
Some ideas are provided in the Deploy a Python Cloud Function with all package dependencies SO question.
Related
I'm trying to run my tests using python -m pytest but I get an error that
ModuleNotFoundError: No module named 'sample'
When using nosetests or anything else it works fine, but when trying to use pytest, it doesn't work.
My tree looks like below, do you have any advice why it doesn't work?
├── LICENSE.txt
├── README.md
├── data
│ └── data_file
├── exported_register.csv
├── pyproject.toml
├── requirements.txt
├── setup.cfg
├── setup.py
├── src
│ └── sample
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-39.pyc
│ │ ├── dziennik.cpython-39.pyc
│ │ ├── przedmiot.cpython-39.pyc
│ │ ├── simple.cpython-39.pyc
│ │ └── uczen.cpython-39.pyc
│ ├── dziennik.py
│ ├── package_data.dat
│ ├── przedmiot.py
│ ├── simple.py
│ └── uczen.py
├── tests
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-39.pyc
│ │ ├── test_ASSERTPY_uczen.cpython-39-pytest-6.2.1.pyc
│ │ ├── test_ASSERTPY_uczen.cpython-39-pytest-6.2.5.pyc
│ │ ├── test_ASSERTPY_uczen.cpython-39.pyc
│ │ ├── test_PYHAMCREST_uczen.cpython-39-pytest-6.2.1.pyc
│ │ ├── test_PYHAMCREST_uczen.cpython-39-pytest-6.2.5.pyc
│ │ ├── test_PYHAMCREST_uczen.cpython-39.pyc
│ │ ├── test_UNITTEST_register.cpython-39-pytest-6.2.1.pyc
│ │ ├── test_UNITTEST_register.cpython-39-pytest-6.2.5.pyc
│ │ ├── test_UNITTEST_register.cpython-39.pyc
│ │ ├── test_UNITTEST_uczen.cpython-39-pytest-6.2.1.pyc
│ │ ├── test_UNITTEST_uczen.cpython-39-pytest-6.2.5.pyc
│ │ ├── test_UNITTEST_uczen.cpython-39.pyc
│ │ ├── test_simple.cpython-39-pytest-6.2.1.pyc
│ │ ├── test_simple.cpython-39-pytest-6.2.5.pyc
│ │ └── test_simple.cpython-39.pyc
│ ├── test_ASSERTPY_uczen.py
│ ├── test_PYHAMCREST_uczen.py
│ ├── test_UNITTEST_register.py
│ ├── test_UNITTEST_uczen.py
│ └── test_simple.py
└── tox.ini
When you run pytest with python -m pytest it uses the current directory as it its working dir, which doesn't contain the sample module (located inside ./src). The way I deal with this is I have a conftest.py inside my tests directory where I add my source dir to python path something like this:
import sys
from pathlib import Path
source_path = Path(__file__).parents[1].joinpath("src").resolve()
sys.path.append(str(source_path))
I've recently started using pytorch and have had similar problems. Couple steps come to mind:
How are you writing the .py file that contains the tests? It may simply be that you need to change up how you import sample within the unit test file. I would expect that you need something like import src.sample.simple. In other words, could be just a pathing issue.
Try a (much) simpler folder structure and try again. If that doesn't work, try to just copy an example of a simple scheme that someone has posted. That is, just get python -m pytest to run somehow, then start slowly adding the complexities of your project.
I have an IoT device (Android device) and some python script to interact with gRPC server running on the same device (as an Android app). I've cross-compiled Python3 and in general it's working - i was able to run some python script with it from Android app (using Runtime.getRuntime().exec(...) and passing PATH, PWD, PYTHONHOME, PYTHONPATH, ANDROID_DATA, ANDROID_ROOT env vars).
The python script that uses gRPC looks as follows:
...
import gprc
...
channel = grpc.insecure_channel(url)
When the script is executed i get the following error:
import error: No module named 'grpc'
Here is the structure of python directories on IoT device (that i've prepared):
├── bin
├── lib
│ ├── pkgconfig
│ ├── python3.9
│ │ ├── asyncio
│ │ ├── collections
│ │ ├── concurrent
│ │ ├── ctypes
│ │ ├── curses
│ │ ├── dbm
│ │ ├── distutils
│ │ ├── encodings
│ │ ├── ensurepip
│ │ ├── html
│ │ ├── http
│ │ ├── idlelib
│ │ ├── importlib
│ │ ├── json
│ │ ├── lib-dynload
│ │ ├── lib2to3
│ │ ├── logging
│ │ ├── multiprocessing
│ │ ├── pydoc_data
│ │ ├── site-packages
│ │ ├── sqlite3
│ │ ├── tkinter
│ │ ├── turtledemo
│ │ ├── unittest
│ │ ├── urllib
│ │ ├── venv
│ │ ├── wsgiref
│ │ ├── xml
│ │ └── xmlrpc
│ └── site-packages
│ ├── google
│ ├── grpc
│ ├── grpcio-1.30.0.dist-info
│ └── protobuf-3.12.2.dist-info
└── share
├── man
│ └── man1
└── terminfo
...
As you can see i've put relevant packages to site-packages (by just copying the same files from my mac machine to iot device which can be incorrect).
What's the right way to do it (where and what exactly should i put relevant libs to python dirs tree)? Can i put any dirs/files to the same dir where the script is (locally)? Is there any lite gRPC impl (probably with limited functionality) in python which can be easily prepared for distribution (eg. copy/pasted)?
FYI I've tried to use python -m pip install grpcio --target and then python -m zipapp resources -m "grpc_serial:main" but it's not working even locally because of module cygrpc not found too (but working if using grpc package that is installed globally):
import error: cannot import name 'cygrpc' from 'grpc._cython' (../python3/lib/python3.9/grpc/_cython/init.py)
If i run "python -m pip install cygrpc --target resources" to have a standalone dist for cygrpc i get ~30 directories (probably transitive deps) about 50Mb which is just crazy heavy.
I can provide a tree output for site-packages if it helps.
Solved it (workaround) by using Thrift RPC which seems to be more lightweight on python side (it required only thrift and six deps installed locally (in the same directory)).
I guess the rootcause for non using of gRPC was not installed transitive deps (not sure it is gRPC or pip issue).
I have been chasing my tail for the last 4 hours here and can't find the solution.
I have the following module/package structure for my project.
.
├── app-cli.py
├── tools
│ ├── __init__.py
│ ├── adapters
│ │ ├── __init__.py
│ │ ├── cli.py
│ │ ├── web.py
│ ├── utils
│ │ ├── __init__.py
│ │ ├── core.py
│ │ │ ├── my_public_method()
│ │ ├── io.py
│ │ │ ├── some_other_public_method()
What I'm trying to do is bundle everything inside utils within the utils name space.
So when I do import tools at the main level, I can access the util functions as:
tools.utils.my_public_method()
tools.utils.some_other_public_method()
Instead of:
tools.utils.core.my_public_method()
tools.utils.io.some_other_public_method()
I have been editing the __init__.py messing around with the levels of imports, attempting to create a shortcut but with no success.
In your __init__.py inside the utils package you can add
from .core import my_public_method
from .io import some_other_public_method
and then you can do:
import tools.utils
tools.utils.my_public_method()
tools.utils.some_other_public_method()
I am developing a Keras environment to train DL models, and I am struggling trying to decide the file structure of the project.
Currently it looks like this:
.
├── build_model.py
├── ctalearn
│ ├── data
│ │ ├── data_loading.py
│ │ ├── data_processing.py
│ │ ├── image_mapping.py
│ │ ├── __init__.py
│ │ └── pixel_pos_files
│ │ ├── FACT_pos.npy
│ │ ├── HESS-II_pos.npy
│ │ ├── HESS-I_pos.npy
│ │ ├── LST_pos.npy
│ │ ├── MSTF_pos.npy
│ │ ├── MSTN_pos.npy
│ │ ├── MSTS_pos.npy
│ │ ├── SST1_pos.npy
│ │ ├── SSTA_pos.npy
│ │ └── SSTC_pos.npy
│ ├── __init__.py
│ ├── predict.py
│ ├── train.py
│ └── utils.py
├── example_model_schematics.yaml
├── example_train_config.yaml
├── models
│ ├── build_cnn_rnn.py
│ └── cnn_rnn.yml
├── setup.py
└── test.py
train.py and predict.py are the main scripts. They take a path to a model architecture stored in .yml format and a configuration file in .yml format as well and run the thing.
data_loading.py is a factory of Sequence objects that are fed to model.fit_generator() / a manager to access the real data files (which are located outside the project structure).
The rest of the datafolder are helper classes for the factory.
The Keras architectures are currently build from a .yaml schematic by the script build_model.py or by a ad hoc script (for example build_cnn_rnn.py) and are stored in .yml format, to be loaded by train.py or predict.py and afterwards compiled and fed a data generator.
Does my structure make sense?
Where should I put build_model.py?
Should I have separate train and predict scripts or mix them together and let the user choose if he want to predict or train via the config.yml file?
Should the model path be specified as a command line argument or as an option inside the config.yaml file?
Should I remove the __init__.py in the inner folder?
When I run an import in my development environment, I seem to get the wrong module.
How do I make sense of this?
$ python -c "import breathe.renderer.rst.doxygen.compound as com; print com"
<module 'breathe.parser.doxygen.compound' from 'breathe/parser/doxygen/compound.pyc'>
The layout of the breathe directory is:
breathe
├── directives.py
├── finder
│ ├── doxygen
│ │ ├── base.py
│ │ ├── compound.py
│ │ ├── index.py
│ │ └── __init__.py
│ └── __init__.py
├── __init__.py
├── nodes.py
├── parser
│ ├── doxygen
│ │ ├── compound.py
│ │ ├── compoundsuper.py
│ │ ├── index.py
│ │ ├── indexsuper.py
│ │ ├── __init__.py
│ ├── __init__.py
├── process.py
├── renderer
│ ├── __init__.py
│ └── rst
│ ├── doxygen
│ │ ├── base.py
│ │ ├── compound.py
│ │ ├── domain.py
│ │ ├── filter.py
│ │ ├── index.py
│ │ ├── __init__.py
│ │ └── target.py
│ ├── __init__.py
└── transforms.py
Check your breathe/renderer/rst/__init__.py file. I suspect that you have something like
from breathe.parser import doxygen
What happens is that when a module is imported, the module's __init__.py is executed. Then the next submodule is imported and it's __init__.py is run.
So, in the above situation, you'll get breathe.renderer.rst importing fine, but when running
breathe.renderer.rst.__init__, the breathe.parser.doxygen module is bound to the local name doxygen overriding the doxygen submodule. So you should see something like this:
>>> from breathe.renderer import rst
>>> print(rst.doxygen)
<module 'breathe.parser.doxygen' from 'breathe/parser/doxygen/__init__.pyc'>
Possible solutions:
Use absolute imports in __init__.py
Keep as much code as possible out of __init__.py
Use import inside the functions it's needed rather than globally. (This can also resolve circular import problems)
Note that there is nothing that prohibits code in __init__.py, and several modules in the stdlib do it too. However, because of exactly this sort of issue it is generally considered a good idea to avoid doing it too much.