I want to create a multi-serivce app engine app, using the first diagram shown on this page.
https://cloud.google.com/appengine/docs/standard/python/configuration-files#an_example
I want to use third party libraries so I used have a lib folder under the root directory, one yaml under the root directory.
Then I want one microservice called predict. So I created a folder called predict under the root directory as well, then under this folder, I want to write py files using the packages in the lib as well.
What I'm doing is like this:
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
vendor.add(os.path.join(parentdir, 'lib'))
Which didn't work for me as the error says:
ValueError: virtualenv: cannot access /base/data/home/apps/f~project-name/lib: No such virtualenv or site directory
Obviously it didn't work for me, what's the right way to do this?
Update: Dan's solution worked for me! So when I'm deploying my microservice predict, I need to go inside predict directory and deploy it. I think that's why it can't find lib. By symlink the lib library using bash.
ln -s ../lib/ ./lib
I solved this problem.
The problem you're facing is that each service can only access files in its own service directory, which is the directory where its app.yaml file is located, they can't access the app's root directory.
My approach (all my services use standard environment):
created a lib dir in the top app dir and installed in it each package that I want shared across multiple services
each service has its own subdir in the app's dir and a lib directory in it where:
I installed libraries needed only by that service (if any)
I symlinked the needed shared libraries from the top lib dir
This is my app dir structure:
app_dir/
app_dir/dispatch.yaml
app_dir/cron.yaml
app_dir/index.yaml
app_dir/queue.yaml
app_dir/lib/
app_dir/lib/shared_lib_1/
app_dir/lib/shared_lib_2/
app_dir/lib/shared_lib_3/
app_dir/service1/
app_dir/service1/app.yaml
app_dir/service1/lib/shared_lib_1 -> ../../lib/shared_lib_1
app_dir/service1/lib/shared_lib_2 -> ../../lib/shared_lib_2
app_dir/service1/lib/service1_lib_1
app_dir/service2/
app_dir/service2/app.yaml
app_dir/service2/lib/shared_lib_2 -> ../../lib/shared_lib_2
app_dir/service2/lib/shared_lib_3 -> ../../lib/shared_lib_3
app_dir/service2/lib/service2_lib_1
No fumbling with the lib path is required, all I have in each service is
vendor.add('lib')
See related:
How do I access a vendored library from a module in Python Google App Engine?
Can a default service/module in a Google App Engine app be a sibling of a non-default one in terms of folder structure?
Check all the directories in the path and the final file to make sure they all exist first before code runs. If they don't, prepend the code you showed here with code to create them...
Related
I am experimenting with Google Cloud Platform buildpacks, specifically for Python. I started with the Sample Functions Framework Python example app, and got that running locally, with commands:
pack build --builder=gcr.io/buildpacks/builder sample-functions-framework-python
docker run -it -ePORT=8080 -p8080:8080 sample-functions-framework-python
Great, let's see if I can apply this concept on a legacy project (Python 3.7 if that matters).
The legacy project has a structure similar to:
.gitignore
source/
main.py
lib
helper.py
requirements.txt
tests/
<test files here>
The Dockerfile that came with this project packaged the source directory contents without the "source" directory, like this:
COPY lib/ /app/lib
COPY main.py /app
WORKDIR /app
... rest of Dockerfile here ...
Is there a way to package just the contents of the source directory using the buildpack?
I tried to add this config to the project.toml file:
[[build.env]]
name = "GOOGLE_FUNCTION_SOURCE"
value = "./source/main.py"
But the Python modules/imports aren't set up correctly for that, as I get this error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
Putting both main.py and /lib into the project root dir would make this work, but I'm wondering if there is a better way.
Related question, is there a way to see what project files are being copied into the image by the buildpack? I tried using verbose logging but didn't see anything useful.
Update:
The python module error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
was happening because I moved the lib dir into source in my test project, and when I did this, Intellij updated the import statement in main.py without me catching it. I fixed the import, then applied the solution listed below and it worked.
I had been searching the buildpack and Google cloud function documentation, but I discovered the option I need on the pack build documentation page: option --path.
This command only captures the source directory contents:
pack build --builder=gcr.io/buildpacks/builder --path source sample-functions-framework-python
If changing the path, the project.toml descriptor needs to be in that directory too (or specify with --descriptor on command line).
Here's what I have:
Project A
----data_I_need.p
Project B
----use_data.py
If data_I_need.p were a source file (and in my path), I could easily import it into Project B. Does such method exist for non-source files?
Right now, I'm having to accomplish this with
with open('C:/......./data_I_need.p', 'rb') as data:
Project A is in my path (defined as 'Content Root' in PyCharm). I can imput source files from Project A.
To recap, the problem is that any Python file in PYTHONPATH can be imported, but resources (non-python files) cannot be loaded without an explicit path.
Project_A
----some_module.py
----data_I_need.p
Project_B
----use_data.py
Here is a solution. In Project B/use_data.py, include the following.
import os
import pathlib
import pickle
# full path to Project_B
here = os.path.dirname(__file__)
# path to folder *above* Project_B ---- joined to Project_A
PROJECT_A = os.path.join(*pathlib.Path(here).parts[:-1], 'Project A')
with open(os.path.join(PROJECT_A, 'data_i_need.p')) as resource:
var = pickle.load resource
This will work even if you're working in a shared drive (e.g., OneDrive) from two different operating systems.
Optionally, you could just set up a "shared_resources" folder and include a python script in that folder to access the resources, again working from os.path.dirname(__file__) then navigating the relative path.
I'm working on a Flask extension from which I want to create a directory in the project's root path on the file system.
Suppose we have this directory structure
/project
/app
/tests
/my_folder
manage.py
my_folder should be created dynamically by the extension, which is a test utility and wraps the application under test in the /tests directory. However, I'm struggling to determine the project's root path within my extension.
For now, I am trying to guess the path from the run file:
def root_path(self):
# Infer the root path from the run file in the project root (e.g. manage.py)
fn = getattr(sys.modules['__main__'], '__file__')
root_path = os.path.abspath(os.path.dirname(fn))
return root_path
This obviously breaks as soon as the tests are run from within the IDE instead of the manage.py. I could simply infer the project's root relative to the app or tests directory, but I don't want to make any assumptions regarding the name or structure of these directories (since multiple apps might be hosted as subpackages in a single package).
I was wondering if there is a best practice for this type of problem or an undocumented method which the Flask object provides (such as get_root_path).
app.root_path contains the root path for the application. This is determined based on the name passed to Flask. Typically, you should use the instance path (app.instance_path) not the root path, as the instance path will not be within the package code.
filename = os.path.join(app.instance_path, 'my_folder', 'my_file.txt')
app.root_path is the absolute path to the root directory containing your app code.
app.instance_path is the absolute path to the instance folder. os.path.dirname(app.instance_path) is the directory above the instance folder. During development, this is next to or the same as the root path, depending on your project layout.
I'm following this tutorial
https://developers.google.com/bigquery/articles/dashboard#downloadinstalllibraries
Here they say that
cd source/
enable-app-engine-project hellodashboard/
which will copy some 60 files into the project directory.But i saw only some 10 files copied.And the oauth2client folder in project directory only has __init__.pyfile
Since i'm using
from oauth2client.appengine import oauth2decorator_from_clientsecrets
When i run my program it shows an error
from oauth2client.appengine import oauth2decorator_from_clientsecrets
ImportError: No module named appengine
So i manually copied all files from oauth2client zip to the project oauth2client directory.
Now when i run my program it doesn't show any error and seems to run fine.Is it some thing messy with the enable-app-engine-project command or am i doing some thing wrong.
enable-app-engine-project tries to locate sources based on the path(s) where the modules are installed on your local system. Have you installed these modules locally?
As an alternative, you can simply copy the following directories into your App Engine project manually:
SOURCES = [
'gflags',
'gflags_validators',
'httplib2',
'oauth2client',
'oauth2',
'apiclient',
'uritemplate',
]
I plan to organize my python project the following way:
<my_project>/
webapp/
mymodulea.py
mymoduleb.py
mymodulec.py
mylargemodule/
__init.py__
mysubmodule1.py
mysubmodule2.py
backend/
mybackend1.py
mybackend2.py
lib/
python_external_lib1.py
python_external_large_lib2/
__init__.py
blabla.py
python_external_lib2.py
in my development IDE (PYdev) to have all working I have setup webapp/, backend/ and lib/ as source folders and all of course works.
How can I deploy it on a remote server? Have I to set PYTHONPATH in a startupscript ?Or have I to it programmatively?
If you are treating webapp, backend, and lib as source folders, then you are importing (for example) mymodulea, mybackend1, and python_external_large_lib2.
Then on the server, you must put webapp, backend, and lib into your python path. Doing it in some kind of startup script is the usual way to do it. Doing it programmatically is complicated because now your code needs to know what environment it's running in to configure the path correctly.