We Keep Coding

Google Cloud Buildpack custom source directory for Python app

I am experimenting with Google Cloud Platform buildpacks, specifically for Python. I started with the Sample Functions Framework Python example app, and got that running locally, with commands:
pack build --builder=gcr.io/buildpacks/builder sample-functions-framework-python
docker run -it -ePORT=8080 -p8080:8080 sample-functions-framework-python
Great, let's see if I can apply this concept on a legacy project (Python 3.7 if that matters).
The legacy project has a structure similar to:
.gitignore
source/
main.py
lib
helper.py
requirements.txt
tests/
<test files here>
The Dockerfile that came with this project packaged the source directory contents without the "source" directory, like this:
COPY lib/ /app/lib
COPY main.py /app
WORKDIR /app
... rest of Dockerfile here ...
Is there a way to package just the contents of the source directory using the buildpack?
I tried to add this config to the project.toml file:
[[build.env]]
name = "GOOGLE_FUNCTION_SOURCE"
value = "./source/main.py"
But the Python modules/imports aren't set up correctly for that, as I get this error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
Putting both main.py and /lib into the project root dir would make this work, but I'm wondering if there is a better way.
Related question, is there a way to see what project files are being copied into the image by the buildpack? I tried using verbose logging but didn't see anything useful.
Update:
The python module error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
was happening because I moved the lib dir into source in my test project, and when I did this, Intellij updated the import statement in main.py without me catching it. I fixed the import, then applied the solution listed below and it worked.

I had been searching the buildpack and Google cloud function documentation, but I discovered the option I need on the pack build documentation page: option --path.
This command only captures the source directory contents:
pack build --builder=gcr.io/buildpacks/builder --path source sample-functions-framework-python
If changing the path, the project.toml descriptor needs to be in that directory too (or specify with --descriptor on command line).

Google App Engine - Multiple yaml files in one project share a lib

I want to create a multi-serivce app engine app, using the first diagram shown on this page.
https://cloud.google.com/appengine/docs/standard/python/configuration-files#an_example
I want to use third party libraries so I used have a lib folder under the root directory, one yaml under the root directory.
Then I want one microservice called predict. So I created a folder called predict under the root directory as well, then under this folder, I want to write py files using the packages in the lib as well.
What I'm doing is like this:
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
vendor.add(os.path.join(parentdir, 'lib'))
Which didn't work for me as the error says:
ValueError: virtualenv: cannot access /base/data/home/apps/f~project-name/lib: No such virtualenv or site directory
Obviously it didn't work for me, what's the right way to do this?
Update: Dan's solution worked for me! So when I'm deploying my microservice predict, I need to go inside predict directory and deploy it. I think that's why it can't find lib. By symlink the lib library using bash.
ln -s ../lib/ ./lib
I solved this problem.

The problem you're facing is that each service can only access files in its own service directory, which is the directory where its app.yaml file is located, they can't access the app's root directory.
My approach (all my services use standard environment):
created a lib dir in the top app dir and installed in it each package that I want shared across multiple services
each service has its own subdir in the app's dir and a lib directory in it where:
I installed libraries needed only by that service (if any)
I symlinked the needed shared libraries from the top lib dir
This is my app dir structure:
app_dir/
app_dir/dispatch.yaml
app_dir/cron.yaml
app_dir/index.yaml
app_dir/queue.yaml
app_dir/lib/
app_dir/lib/shared_lib_1/
app_dir/lib/shared_lib_2/
app_dir/lib/shared_lib_3/
app_dir/service1/
app_dir/service1/app.yaml
app_dir/service1/lib/shared_lib_1 -> ../../lib/shared_lib_1
app_dir/service1/lib/shared_lib_2 -> ../../lib/shared_lib_2
app_dir/service1/lib/service1_lib_1
app_dir/service2/
app_dir/service2/app.yaml
app_dir/service2/lib/shared_lib_2 -> ../../lib/shared_lib_2
app_dir/service2/lib/shared_lib_3 -> ../../lib/shared_lib_3
app_dir/service2/lib/service2_lib_1
No fumbling with the lib path is required, all I have in each service is
vendor.add('lib')
See related:
How do I access a vendored library from a module in Python Google App Engine?
Can a default service/module in a Google App Engine app be a sibling of a non-default one in terms of folder structure?

Check all the directories in the path and the final file to make sure they all exist first before code runs. If they don't, prepend the code you showed here with code to create them...

Google App Engine Locally: ImportError: No module named google.cloud.bigquery

As the title says.
So I've added the following to appengine_config.py with no luck:
# appengine_config.py
from google.appengine.ext import vendor
# Add any libraries install in the "lib" folder.
vendor.add('lib')
I did a print sys.path and verified the lib dir contains google/cloud/bigquery
I can import it if I run python myself:
from google.cloud import bigquery
print bigquery.__path__
['/usr/local/lib/python2.7/dist-packages/google/cloud/bigquery']
From a Google App Engine end points:
import google.cloud;print google.cloud.__path__
['/usr/local/lib/python2.7/dist-packages/google/cloud']
Big Query is at that system location. /usr/local/lib/python2.7/dist-packages/google/cloud/biqguery/ exists. However if I try the following from app engine end point:
from google.cloud import bigquery
ImportError: No module named google.cloud.bigquery
The file in question includes and that does not appear to help:
from __future__ import absolute_import
Update
I setup a venv and install everything like this: pip install -t lib/ -r requirements.txt --upgrade. From there if I try import google; print google.__path__ I get:
['lib/google', '/usr/lib/google-cloud-sdk/platform/google_appengine/google']
I think the second path might be the cause.
I've reviewed the following with no success:
Error importing Google Cloud Bigquery api module in python app
https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2366
How to import BigQuery in AppEngine for Python

You need to install google-cloud-bigquery into you app's lib dir, that's where the development server is looking at, not on your system's libs. From Installing a third-party library:
Create a directory to store your third-party libraries, such as lib/.
mkdir lib
Use pip (version 6 or later) with the -t <directory> flag to copy the libraries into the folder you created in the previous
step. For example:
pip install -t lib/ <library_name>

External lib not found when deploy on Google App Engine

I am using a couple of google libs in order to authenticate with firebase in a Python + GAE app.
I have configured the requirements.txt with the following content:
google-auth==1.0.1
requests==2.14.2
requests-toolbelt==0.7.1
This is what I am importing:
import google.auth.transport.requests
When I run pip install, they do get installed locally and I get no errors.
local libs screenshot
But when I try to deploy this application to Google App Engine, all those external libs get the same errors. GAE doesn't find the files:
ImportError: No module named auth.transport.requests

You need to provide your library directory to the google.appengine.ext.vendor.add() method.
Create a file named appengine_config.py in the same folder as your app.yaml file.
Edit the appengine_config.py file and provide your library directory to the vendor.add() method.
# appengine_config.py
from google.appengine.ext import vendor
# Add any libraries installed in the "lib" folder.
vendor.add('lib')
https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27#installing_a_third-party_library

importing from another folder in virtualenv

I'm following the Flask Mega Tutorial, and I'm running into an issue once I get to the second part and restructure my folder structure to match theirs, I cannot import Flask.
My current folder structure is as follows
/FlaskTest
/app
/static, templates etc
/flask
/virtualenv folders etc
/tmp
run.py
as far as I can tell, the folder structures are identical other than naming of the top level directory.
in my __init__.py file (/app/__init__.py), I'm doing as instructed in the tutorial,
from flask import Flask
app = Flask(__name__)
from app import views
I'm getting an Import Error that "cannot import name 'Flask'". I'm guessing the issue is because the flask package was installed to /flask/lib/site-packages.
My question: How can I reference the sub folder of flask/site-packages?
I've read through the python import system documentation and from what I can make of it through the first pass of reading it over, I would need to likely do something like from flask import flask.Flask or something to that effect.
UPDATE: So after cd'ing around the directory and checking pip list, I realized that flask wasn't accessible to my app directory. I ran pip install flask in the app directory. Now my site runs, but I'm not sure if this is the best practice of doing things with Python. Please provide some clarity as what the best practice is for installing packages and where the packages reside.
UPDATE 2: After creating a directory called standalone. In this folder, I created a virtual environment called standalone-test. Once, I did that, I also mkdir'ed app and copied it's contents from FlaskTest so that way the code would be identical. I was able to run the run.py script by using python run.py, but I can't run python -m app like you had said without running into an error. The error is as follows if it helps.
"No module name app.main; 'app' is a package and cannot be directly executed.
I am able to run python run.py as I mentioned, but I'm not able to run the python -m app command as you had mentioned

I think something went wrong in your execution environment. Here are some explanations.
The virtualenv
See the documentation of virtualenv
If you have followed the tutorial:
The flask directory is your virtualenv,
On posix system, you have a flask/bin subdirectory, or
On Windows system, you have a flask\Scripts subdirectory.
I make the assumption that you are on posix system.
To activate your virtualenv, run:
source flask/bin/activate
Your prompt should change to something like: (flask)$.
To list the installed libraries use pip:
pip list
Make sure you see Flask. The tutorial encourages you to install a lot of Flask plugins, so there are a lot of Flask-Something…
If Flask is missing, install it:
pip install Flask
Run your app
Your application is in the app directory, it has an __init__.py file (it's a Python package).
In this file, you have:
from flask import Flask
app = Flask(__name__)
from app import views
From your FlaskTest/ directory, try to run this script like this:
cd FlaskTest/ # if not in this directory
python -m app
This should import Flask, instanciate your app (but don't run it), import the views module.
If app/views.py exist you should have no error.
=> at this point, we have simulated what run.py imports…
Now write run.py in your FlaskTest/ directory:
#!flask/bin/python
from app import app
app.run(debug=True)
Run it like this:
python run.py
Note that the shebang #!flask/bin/python is unusual, but should work in the context of the tutorial.
This should start your http server…