We Keep Coding

Google App Engine - Multiple yaml files in one project share a lib

I want to create a multi-serivce app engine app, using the first diagram shown on this page.
https://cloud.google.com/appengine/docs/standard/python/configuration-files#an_example
I want to use third party libraries so I used have a lib folder under the root directory, one yaml under the root directory.
Then I want one microservice called predict. So I created a folder called predict under the root directory as well, then under this folder, I want to write py files using the packages in the lib as well.
What I'm doing is like this:
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
vendor.add(os.path.join(parentdir, 'lib'))
Which didn't work for me as the error says:
ValueError: virtualenv: cannot access /base/data/home/apps/f~project-name/lib: No such virtualenv or site directory
Obviously it didn't work for me, what's the right way to do this?
Update: Dan's solution worked for me! So when I'm deploying my microservice predict, I need to go inside predict directory and deploy it. I think that's why it can't find lib. By symlink the lib library using bash.
ln -s ../lib/ ./lib
I solved this problem.

The problem you're facing is that each service can only access files in its own service directory, which is the directory where its app.yaml file is located, they can't access the app's root directory.
My approach (all my services use standard environment):
created a lib dir in the top app dir and installed in it each package that I want shared across multiple services
each service has its own subdir in the app's dir and a lib directory in it where:
I installed libraries needed only by that service (if any)
I symlinked the needed shared libraries from the top lib dir
This is my app dir structure:
app_dir/
app_dir/dispatch.yaml
app_dir/cron.yaml
app_dir/index.yaml
app_dir/queue.yaml
app_dir/lib/
app_dir/lib/shared_lib_1/
app_dir/lib/shared_lib_2/
app_dir/lib/shared_lib_3/
app_dir/service1/
app_dir/service1/app.yaml
app_dir/service1/lib/shared_lib_1 -> ../../lib/shared_lib_1
app_dir/service1/lib/shared_lib_2 -> ../../lib/shared_lib_2
app_dir/service1/lib/service1_lib_1
app_dir/service2/
app_dir/service2/app.yaml
app_dir/service2/lib/shared_lib_2 -> ../../lib/shared_lib_2
app_dir/service2/lib/shared_lib_3 -> ../../lib/shared_lib_3
app_dir/service2/lib/service2_lib_1
No fumbling with the lib path is required, all I have in each service is
vendor.add('lib')
See related:
How do I access a vendored library from a module in Python Google App Engine?
Can a default service/module in a Google App Engine app be a sibling of a non-default one in terms of folder structure?

Check all the directories in the path and the final file to make sure they all exist first before code runs. If they don't, prepend the code you showed here with code to create them...

How to run a python program like pycharm does

I've been developing a project in python for some time using pycharm. I have no problem running it in pycharm, but I want to try and running it from the command line (I'm using windows).
When I try to run my main file using python <filename> from within the root directory, I get a module not found error. What is pycharm doing/how can i replicate it?
Also, pycharm creates pycache folders. If my understanding is correct its compiling the files, and that makes the runtime faster. Since my runtime is already long i'd like to do the same.
I'm using python 3.6
edit
File Structure
-Root\
----scheduler\
------main.py
------matrices
------models
------rhc
------Singleton.py
------utils.py
------__init__.py
------apis\
acedb.py
metrics.py
__init__.py
------matrices\
distancematrix.py
__init__.py
------models\
branch.py
constants.py
customer.py
dispatch.py
technician.py
workorder.py
__init__.py
------rhc\
pathfinder.py
rhc.py
schedule.py
sched_time.py
tech_schedule.py
__init__.py
Edit 2
Found the solution, i had to move my main files outside of the modules
Thanks!

If you have the below folder structure for instance. add an empty file named init.py and import from your app.py, in case you have a Module1Class , you can always import it like this
from modules.module1 import Module1Class
from modules.module2 import Module2Class
Folder structure
/app.py
/modules
/module1.py
/module2.py
/__init__.py
The first time you rum app.py from terminal like python app.py, the .pyc files will be created, leaving you with the following folder structure
/app.py
/modules
/module1.py
/module1.pyc
/module2.py
/module2.pyc
/__init__.py
Please refer to the python documentation as it's very well documented on how to create modules and importing them. I'm more used to python2.7 , so my answer might not be an exact fit to newer python versions.
https://docs.python.org/3/tutorial/modules.html
From there you can learn more about __ init __.py and module creation , exporting and importing
PS: I use only text editors to develop in python, as I find pycharm a bit on the heavy side, so I cannot explain how exactly pycharm works behind the curtains.

see ModuleNotFoundError: What does it mean __main__ is not a package? for a good example of ModuleNotFoundError description (was answered fast)
I bet that pycharm is configured to use a different python interpreter of virtual environment.

Python absolute import not working on Ubuntu server

My project has the following structure:
server/ (root of project)
|
|--- __init__.py
|--- requirements.txt
|--- env/ (virtual environment)
|--- app/ (main app folder)
|--- __init__.py (defines a 'app = Flask(__name__)' object)
|--- app.py (runs app on local server)
|--- models.py
|--- views.py
The way I import different modules in app.py on my local machine is do:
# /server/app/app.py
from server.app import app
from server.app.models import *
from server.app.views import *
It works fine on my local machine (using PyCharm IDE, and the Python binary inside virtual environment folder /server/env/bin/.
However, when I push this to the production server running Ubuntu, where I install all dependencies globally, it keeps throwing the error no module named server.app when I run:
python server/app/app.py
Does anyone know why?

Any IDE environment usually sets the pythonpath. E.g. in eclipse right click your project and see the properties. You will see that your main project is listed in the pythonpath. This path is used to locate modules.
Now in your production code you are not in your IDE. So normal python interpreter cannot find your path. Hence you need to specify this.
One way is to add sys.path.append('/path/to/the/project') before you do your import (This should be done for the first script that got executed, in this case app.py, that way you only need to do this once).
You can also add your path permanently to your production environment. See this post.

As #Rash mentionned, your IDE very probably add the directory containing /server to your python path. You can check this in your app.py by adding
import sys
print "\n".join(sys.path)
before your imports.
When you're manually running your app, ie python server/app/app.py, the parent of your server directory is obviously not in your python path, so you have to add yourself.

Structure Python application with client and server parts

I have an application that is currently in the following folder structure:
myapp/
client/
core/
server/
template_files/
It has a server side and a client side component and when I deploy the code to a user, I don't want to include the server side code as well. Both client and server need the core code to run.
Reading about general Python project structure, I realise I should start by changing my structure to:
myapp/
myapp/
client/
core/
server/
template_files/ (template is only needed by the server)
bin/
setup.py
What's the best way to structure my directories and do code deployment?

You might want to have three separate packages instead of one, even your second structure will result in a package (python egg) that contains all the sub-modules which will result in the server code being packaged with the client code.
What you want is to split them up into three individual packages (i.e. myapp.client, myapp.core and myapp.server) and they will have separated directory structures, so in effect you would have something like
myapp.client/
myapp/
client/
setup.py
myapp.core/
myapp/
core/
setup.py
myapp.server/
myapp/
server/
template_files/
setup.py
As they will all become proper python packages, you can define dependencies in the setup.py for myapp.client and myapp.server to require myapp.core, so if/when you deploy the packages onto pypi (or others) your users can simply do pip install myapp.client to get the client library installed onto their system with all dependencies fetched.
You don't necessarily have to have a bin in anywhere. You can take advantage of the entry_points attribute in the setup function to let setuptools create the "binary" for you in an OS agnostic manner. Just define the main function(s) inside your library and let setuptools create the executable for you for your users.
Lastly, you might want to take a look at what other open source projects have done for packaging their libraries, here are some examples:
Flask
Jinja2
plone.app.discussion

Python - packages and settings file

I have a python package that needs to pull in settings from my project directory, here is how my project is currently structured:
~/Project/bin/mypackage
- package files
~/Project/myproject/
- project files
- start.py
- settings.py
I guess it's similar to how Django is structured, you have a settings.py file in your project directory that is somehow referenced by the Django system package in your Python directory.
So, if I am running start.py like so:
python ~/Project/myproject/start.py
..and start.py imports and utilizes the mypackage package, is there any way I can reference the settings.py file local to start.py from within the package? Would I have to load the settings file in start.py and store the values in a global? Does anyone know how this is possible?

The way I see it you have several options:
look for settings and import them either from the current working directory or as determined from environment variables. This is the "django way" using DJANGO_SETTINGS_MODULE and PYTHONPATH. This is nice and magical when it works and inconvenient when it doesn't such as in your case when you are running from a different directory.
rely on module search path which will include the directory of the calling package. Nice and simple but the settings will vary based on the caller. For example all you need in mypackage is:
import settings
pass in settings as a variable

The directory containing the script that was used to invoke the python interpreter is added to the PYTHONPATH. It is available at sys.path[0]. See http://docs.python.org/library/sys.html#sys.path
This means that settings should be available from mypackage.mymodule simply by import settings.
However, I would consider handling the loading of settings in start.py and structuring your app so that a settings object (perhaps just a dict) is passed to it.