We Keep Coding

Google Cloud Buildpack custom source directory for Python app

I am experimenting with Google Cloud Platform buildpacks, specifically for Python. I started with the Sample Functions Framework Python example app, and got that running locally, with commands:
pack build --builder=gcr.io/buildpacks/builder sample-functions-framework-python
docker run -it -ePORT=8080 -p8080:8080 sample-functions-framework-python
Great, let's see if I can apply this concept on a legacy project (Python 3.7 if that matters).
The legacy project has a structure similar to:
.gitignore
source/
main.py
lib
helper.py
requirements.txt
tests/
<test files here>
The Dockerfile that came with this project packaged the source directory contents without the "source" directory, like this:
COPY lib/ /app/lib
COPY main.py /app
WORKDIR /app
... rest of Dockerfile here ...
Is there a way to package just the contents of the source directory using the buildpack?
I tried to add this config to the project.toml file:
[[build.env]]
name = "GOOGLE_FUNCTION_SOURCE"
value = "./source/main.py"
But the Python modules/imports aren't set up correctly for that, as I get this error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
Putting both main.py and /lib into the project root dir would make this work, but I'm wondering if there is a better way.
Related question, is there a way to see what project files are being copied into the image by the buildpack? I tried using verbose logging but didn't see anything useful.
Update:
The python module error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
was happening because I moved the lib dir into source in my test project, and when I did this, Intellij updated the import statement in main.py without me catching it. I fixed the import, then applied the solution listed below and it worked.

I had been searching the buildpack and Google cloud function documentation, but I discovered the option I need on the pack build documentation page: option --path.
This command only captures the source directory contents:
pack build --builder=gcr.io/buildpacks/builder --path source sample-functions-framework-python
If changing the path, the project.toml descriptor needs to be in that directory too (or specify with --descriptor on command line).

load_dotenv fails when running from a package

Using load_dotenv to read python variables from a .env file.
Building a python package using the setup module and packaging it into a command get_values
Cases when code runs successfully
1- Codebase runs and I can read the environment values only if the command 'get_values' is running from the same directory as codebase.
2- Codebase runs as a standalone Python program
Cases, when code doesn't, runs successfully
1- After building the package, I run the commands 'get_values' the code fails to read the environment variables and get the string value as NoneType
Current Codebase hierarchy:
.env file
OIDC_CLIENT_ID='xxxxxxxxxxx'
Code to read environment file
basepath = Path()
basedir = str(basepath.cwd())
envars = basepath.cwd() / 'config.env'
load_dotenv(envars)
Reading the environment variable
print(os.getenv('OIDC_CLIENT_ID'))

Make sure your current working directory contains a 'config.env' file in the cases where it's not working.

Gcloudignore file is not respected during deployment to App Engine

gcloud app deploy keeps uploading all files in source directory, although I have explicitly excluded them using a .gcloudignore file. For example, the virtual environment folder env is uploaded, which is causing an error because the deployment contains more then 10,000 files then.
I am working under Windows 10 with Python 3.7 and gcloud SDK version 251.0.0. I tried both the beta and the normal version of gcloud app deploy.
The .gcloudignore file contains just the following:
.gcloudignore
# If you would like to upload your .git directory, .gitignore file or
# files from your .gitignore file, remove the corresponding line below:
.git
.gitignore
#!include:.gitignore
I can see in the outputs with --verbosity=info flag that it recognized the .gcloudignore file, but then it uploads the env folder to Cloud Storage. I would expect this folder to be skipped. Git works as expected.

You can "include" everything ignored by your .gitignore file in your .gcloudignore file by adding the following line:
#!include:.gitignore
If your .gitignore file is already ignoring the env directory, this will cause gcloud to ignore it as well (and every other file that git is ignoring).

Your .gcloudignore file contains two leading spaces on each line which are not allowed.
These spaces are copied from the documentation so many people will be facing this issue.
See the documentation here (but do not blindly copy from their until they fixed it)

Run Python plugins in Hiero

I am facing a small problem in executing custom made python code from Hiero. As everybody knows, the first place to look for details is the offical documentation "Nukepedia", the Python Dev Guide and the Hiero User Guide. Well according to the Python Dev Guide, Hiero creates a ".hiero" directory by default which lets people add plugin paths to use them in the software. Well, I can't find that directory and I deduced after several tests that Hiero is using the init.py saved in the ".nuke" directory.
So I thought that maybe I could add the plugin paths there but kept getting the famous Import Error for the _fnpython module (before creating Python/Startup directory).
I added Python/Startup folders in .nuke and added the plugins in Startup, I got the same error, I even tried it by adding the path to the plugins in init.py and got the same error too.
Then I created a ".hiero" folder and did the same thing as before but Hiero never took that folder into consideration, I deduced that by printing some strings in the console, Hiero always took the init.py saved in the ".nuke" folder and kept showing the same error.
Finally, I tried to look into the installation process and try to seperate Nuke and Hiero's folders maybe that would create the ".hiero" directory but everything was automated.
The code that I want to run is given by Nuke and Hiero (in the examples directory), I just can't figure out what to do in order to run it from the program.
What should I do in order to fix this problem ?
Thank you in advance.

The setup for The Foundry HIERO is a little different than for NUKE.
HIERO has a core module. You'll see it in __init__.py file:
import FnRedirect
import FnPythonFixes
import core
import ui
try:
if hasattr(core, "taskRegistry"):
import importers
import exporters
except ImportError as e:
pass
I'm running HIERO and NUKE STUDIO on a Mac, so there's a full path to HIERO's __init__.py file inside package contents:
/Applications/Nuke10.5v5/Contents/MacOs/pythonextensions/site-packages/hiero/__init__.py
You need to import this module using:
import hiero.core
or using a reference to the core package:
from core import *
To find HIERO's current paths you have to run this line in its Script Editor:
print hiero.core.pluginPath()
Click this link for further details: Hiero's Environment Setup
All these instructions are suitable for macOS 10.9 and above. Here are two blocks of steps: first for Terminal Mode and second for UI Mode.
BLOCK 1: setup for Terminal Sessions
You need to manually create .hiero directory in your Home area.
The recommended default location for running Python on startup is:
~/.hiero/Python/Startup
~/.hiero/Python/StartupUI
Type in your bash Terminal (when you're inside your Home user directory) the following line:
mkdir .hiero/
then:
mkdir .hiero/Python/
and then:
mkdir .hiero/Python/StartupUI/
then navigate to Home directory with:
cd ~
and check it with:
ls -a
Also you can specify any number of user-defined paths using the environment variable HIERO_PLUGIN_PATH, just like the standard Unix PATH environment variable.
For that you need to set up an environment variable in .bash_profile. To run in Terminal PICO editor just type (remember you need an administrator's password for sudo command):
sudo pico .bash_profile
and paste these three lines in it (change swift for <yourName> and save this file):
echo HIERO environment var is set...
export HIERO_PLUGIN_PATH=/Users/swift/.hiero/Python/StartupUI/
export PATH=$PATH:$HIERO_PLUGIN_PATH
Write out a file with ctrl o
Exit pico editor with ctrl x
Restart Terminal
In Terminal you could print this environment variable typing:
printenv HIERO_PLUGIN_PATH
You should put inside that StartupUI directory menu.py, any set of xxxx.py or xxxx.pyc files, as well as __init__.py file.
Now you can use /Users/swift/.hiero/Python/StartupUI/ path in Terminal Mode.
BLOCK 2: setup for UI Sessions
To assign the current paths that HIERO searches when loading plug-ins, you need to create __init__.py file with the following lines:
import hiero.core
path='/Users/swift/.hiero/Python/Startup/'
hiero.core.addPluginPath(path)
After that make Python/Startup/ subdirectories under ~/.nuke/ folder.
It's not a mistake: I typed .nuke.
Then place this __init__.py file into /Users/swift/.nuke/Python/Startup/ directory.
Restart HIERO (or NUKE STUDIO) if it works.
After that launch HIERO or NUKE STUDIO and run
print hiero.core.pluginPath()
command in the HIERO's Script Editor or in NUKE STUDIO's Script Editor and you'll see this result:
After that you'll find new __init__.pyc file in /Users/swift/.nuke/Python/Startup/ directory.
Each package or module discovered when you launch HIERO is imported and added to the built-in package hiero.plugins.
TEST 1: custom_guides.py
I do not have a commercial version of HIERO so I tested custom_guides.py script ( found here ) using NUKE STUDIO NC.
I placed custom_guides.py in ~/.nuke/Python/Startup directory and then added two lines to NUKE's init.py file located in ~/.nuke directory.
import nuke
nuke.pluginAddPath("./Python/Startup")
The only thing I could say: "it works" Do the same actions as I did and it'll work for HIERO.
Look at safe_zone and masking_ratio dropdown menus. They are different: before and after.
Before uploading custom_guides.py script:
After uploading custom_guides.py script:
# file custom_guides.py contains these lines:
viewer_masks = [
hiero.ui.guides.MaskGuide("NTSC", 0.91),
hiero.ui.guides.MaskGuide("PAL", 1.09),
hiero.ui.guides.MaskGuide("NTSC_16:9", 1.21),
hiero.ui.guides.MaskGuide("PAL_16:9", 1.46),
hiero.ui.guides.MaskGuide("Cinemascope 2:1", 2.0)
]
TEST 2: web_browser.py
I placed web_browser.py file in ~/.nuke/Python/Startup directory. This Python script creates dockable panel with web browser written with PySide Qt.
I do not have a commercial version of HIERO so I tested web_browser.py script ( found here ) using NUKE STUDIO NC.

How to use Travis CI with some files in gitignore?

I have a Flask app that has its configurations in a file called settings.py. I've put this file in .gitignore because the project is in a public repo. Travis-CI was working before I added tests into my project even though settings.py was in .gitignore. After adding tests to the project, the build started failing with the following output:
Debugged import:
- 'settings' not found.
Original exception:
ImportError: No module named 'settings'
My .travis.yml file looks like this:
language: python
python:
- "3.4"
- "3.5"
# command to install dependencies
install:
- pip install -r requirements.txt
# command to run tests
script: python tests.py
Does this mean that in order to use travis-ci, we have to include all necessary files in the repo? Or is there a workaround? The repo on GitHub can be found here.

#dirn's comment of using a default settings.py file and then overriding some settings with encrypted environment variables on Travis is a good idea, certainly worth it if there are only a couple of differences.
However, if you can't be bothered or it's too complicated breaking up your settings, you could install the Ruby Travis command line client gem, which is useful for quite a few things.
With the client on your machine you can use Travis' file encryption feature to encrypt your whole settings.py file, and then commit the encrypted version (which will have an .enc file extension) to GitHub. Travis will then be able to decrypt the file during the CI run, as long as you add the right commands to the .travis.yml file, say in a before_install step. Detailed instructions are on the file encryption page.

I did a trick in .travis.yml
After commit and before Travis build, create the ignored file like this:
before_install:
- cp .ignored.file.copy ignored.file
This way, the build will succeed without the actual gitignore-ed file.