We Keep Coding

Google Cloud Buildpack custom source directory for Python app

I am experimenting with Google Cloud Platform buildpacks, specifically for Python. I started with the Sample Functions Framework Python example app, and got that running locally, with commands:
pack build --builder=gcr.io/buildpacks/builder sample-functions-framework-python
docker run -it -ePORT=8080 -p8080:8080 sample-functions-framework-python
Great, let's see if I can apply this concept on a legacy project (Python 3.7 if that matters).
The legacy project has a structure similar to:
.gitignore
source/
main.py
lib
helper.py
requirements.txt
tests/
<test files here>
The Dockerfile that came with this project packaged the source directory contents without the "source" directory, like this:
COPY lib/ /app/lib
COPY main.py /app
WORKDIR /app
... rest of Dockerfile here ...
Is there a way to package just the contents of the source directory using the buildpack?
I tried to add this config to the project.toml file:
[[build.env]]
name = "GOOGLE_FUNCTION_SOURCE"
value = "./source/main.py"
But the Python modules/imports aren't set up correctly for that, as I get this error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
Putting both main.py and /lib into the project root dir would make this work, but I'm wondering if there is a better way.
Related question, is there a way to see what project files are being copied into the image by the buildpack? I tried using verbose logging but didn't see anything useful.
Update:
The python module error:
File "/workspace/source/main.py", line 2, in <module>
from source.lib.helper import mymethod
ModuleNotFoundError: No module named 'source'
was happening because I moved the lib dir into source in my test project, and when I did this, Intellij updated the import statement in main.py without me catching it. I fixed the import, then applied the solution listed below and it worked.

I had been searching the buildpack and Google cloud function documentation, but I discovered the option I need on the pack build documentation page: option --path.
This command only captures the source directory contents:
pack build --builder=gcr.io/buildpacks/builder --path source sample-functions-framework-python
If changing the path, the project.toml descriptor needs to be in that directory too (or specify with --descriptor on command line).

How to import an old commit of a Python module I wrote myself in a script?

I would like to import an old commit of a Python module (that I wrote myself) in a script.
I've found a solution here. It says to write the following at the top of the file.
%%bash cd <project>_func # move inside the directory where you have the repository you need
git checkout <git commit id> # restore the version of the code you will use
(<project>_func is the name of the directory containing the module I need to import.)
I've tried writing at the top of the script in which I'm importing my module, but it just returns a syntax error. I'm using Spyder, not running the script from a shell.
What is the correct way to implement that?
After #Kavindu Ravishka comment, I'm including all the steps to reproduce the error below:
1°) Directory structure :
├── ModDir
│ └── mod.py
└── ScriptDir
└── script.py
2°) Contents of mod.py :
com_nb = 1 for the first commit
3°) Open shell in ModDir
4°) Type :
git init
git add .
git commit -m 'com_nb = 1'
Then
git log
which returns
commit 1234a # commit id was modified for clarity
Author : me <me#me.me>
Date : Today
com_nb = 1
5°) Create script.py in ScriptDir, with the following contents :
%%bash cd /absolute/path/to/ModDir
git checkout 1234a
from pathlib import Path
import sys
path_root = Path(__file__).parents[1]
sys.path.append(str(pathroot))
from ModDir.mod import com_nb
print(com_nb)
6°) Go back to step 2 and modify the contents of mod.py :
com_nb = 2
7°) Once again, open a shell in ModDir and type:
git add .
git commit -m 'com_nb = 2'
8°) Run script.py in Spyder. My hope would be that it prints 1, the value of com_nb in the first commit. Instead, it returns SyntaxError : invalid decimal literal, pointing to the "1" in 1234a because, I assume, it doesn't interpret the first two lines in script.py as bash commands.

I tried to recreate your Problem. And tried several changes
%%bash cd /tmp
git checkout 1234a
%%bash cd /tmp
#!git checkout 1234a
%%bash cd /tmp
!git checkout 1234a
%%bash cd /tmp
%%git checkout 1234a
All were failed with errors. If you trying to run shell commands via python use os library.
import os
os.system("bash cd /absolute/path/to/ModDir")
os.system("git checkout 1234a")
this would solve the problem

pathlib Path resolves installed path directory of package instead of source code directory

I have packaged my project using setup.py and project folder structure looks like below.
api-automation
api
packagename
__init__.py
user.py
payloads
a.json
b.json
tests
conftest.py
setup.cfg
setup.py
README.rst
I have created virtual environment in below folder with name "myenv_1",
/Users/basavarajlamani/Documents/environments/
and i have installed above repo in this virtual environment.
I tried a lot on stackoverflow and internet but did not found answer.
code of user.py file
from pathlib import Path
current_dir = str(Path(__file__).resolve().parent)
def func():
print("current_dir", current_dir)
code of conftest.py
from packagename.user import func
func()
If I run user.py file directly(python3 user.py), i will get the correct directory path as below,
current_dir /Users/basavarajlamani/Documents/repos/api-automation/api/packagename
But if I run conftest.py file(python3 conftest.py), I am getting installed path as below which i don't want and I want to get directory path like when i run user.py file directly,
current_dir
/Users/basavarajlamani/Documents/environments/myenv_1/lib/python3.7/site-packages/packagename
Please help, how i can solve this problem.

I suspect you didn't use the correct option when bootstrapping your development environment.
Try:
cleanup your development virtualenv or delete it and create a new one.
cd the/root/of/your/source/tree
pip install -e .
The important point is the -e option. Read the pip manual.

Run python script in package

I am struggling with running python script in shell. I use PyCharm where is everything ok, but I want to run script without running PyCharm.
So my project folder is like:
data/
file.txt
main/
__init__.py
script.py
tools/
__init__.py
my_strings.py
I want to run main/script.py, which start with from tools import my_strings and working directory should be data/.
My PyCharm config is:
Script path: <PROJECT>/main/script.py
Working directory: <PROJECT>/data
Add content roots to PYTHONPATH: YES
Add source roots to PYTHONPATH: YES
So I want to run main/script.py in shell on Ubuntu. I tried:
PYTHONPATH=<PROJECT>
cd <PROJECT>/data
python3 ../main/script.py
But I just got: ImportError: No module named 'tools'

Check out this post, it's explains the PYTHONPATH variable.
How to use PYTHONPATH and the documentation the answer points to https://docs.python.org/2/using/cmdline.html#envvar-PYTHONPATH
When you run from the data directory by default python can't find your tools directory.
Also regarding your comment about needing to run from the data directory, you could just use open('../data/file.txt') if you decided to run from the main directory instead.
Ideally, you should be able to run your script from anywhere though. I find this snippet very useful os.path.dirname(sys.argv[0]). It returns the directory in which the script exists.

I simply forgot to export $PYTHONPATH as suggested by Steve.

Can't launch a program from a different directory via Pexpect

I have troubles launching a simple HelloWorld program via Pexpect module.
I have a directory with the HelloWorld binary - hw, expect script - m.py, and a directory with the same script inside.
├── hw
├── m.py
├── main.cpp
└── dir
└── m.py
Here is my expect script:
import pexpect
child = pexpect.spawn("./hw", cwd = /absolute/path/to/parent/directory")
child.expect("!")
print(child.before)
If i run the script from the parent directory, everything works great. However, If I run it from any other directory, like dir here, I get the following error:
pexpect.ExceptionPexpect: The command was not found or was not executable: ./hw.
How do I fight this?
I have tried this on Mac OS and Ubuntu. HelloWorld binary works fine, it is set as executable. Python 2.7.6, pexpect 3.3

To run an executable hw, either its parent directory should be in PATH envvar or you should provide the full path. If the path is relative (not recommended) then it is a path relative to your current working directory regardless of cwd value.
If you want to run hw from its directory:
import os
import pexpect # $ pip install pexpect
hw = '/absolute/path/to/parent/directory/hw'
child = pexpect.spawn(hw, cwd=os.path.dirname(hw))
# ...

#jfs answer is pretty clear, just want to add some more,
If you got an executable script (script.sh) in the directory /path/to/parent/dir
child = pexpect.spawn("./script.sh", cwd="/path/to/parent/dir") # wont work
As with ./ it is trying to execute from the scripts current working directory, if you somehow need that to work then you can add os.chdir(), so
os.chdir("/path/to/parent/dir")
child = pexpect.spawn("./script.sh") # works
If you don't need to change the script path all to run the child process, can spawn the pexpect process with,
child = pexpect.spawn("/path/to/parent/dir/script.sh") # wont works for my script
This won't works if there is relative paths mentioned the script.sh, cz the cwd(current working directory) is the python script's cwd. In my case script.sh needs resources from the directory it exists,
So in order to execute it,
child = pexpect.spawn("/path/to/parent/dir/script.sh", cwd="/path/to/parent/dir") # works
if you don't got the hard coded parent path of script.sh, use os.path.dirname("<absolute path to script.sh>") as mentioned in the previous answer