Cannot use jinja2 PackageLoader on Google App Engine - python

I'm trying to use the PackageLoader jinja2 provides but I cannot get it to work.
In my app.yaml I have the required libraries declared:
libraries:
- name: jinja2
version: latest
- name: setuptools
version: latest
The smallest example I could create:
import webapp2, sys
from jinja2 import Environment, PackageLoader
sys.path += ['lib/somepackage']
class Test(webapp2.RequestHandler):
def get(self):
env = Environment(loader=PackageLoader('common'))
self.response.write(env.get_template('test.html').render())
routes = [(r"/", Test)]
app = webapp2.WSGIApplication(routes, debug=True)
The package common exists in the directory lib/somepackage and has a package templates which contains a file test.html. The html file only contains the text 'test'.
When the PackageLoader is constructed it's provider is always a NullProvider when in Google App Engine. When I manually request a provider I get a DefaultProvider so obviously something is wrong here.
To request a provider I ensure lib/somepackage is in the sys.path by going to that dir, then:
>>> import pkg_resources
>>> provider = pkg_resources.get_provider('common')
>>> provider
<pkg_resources.DefaultProvider instance at 0x8490b2c>
When this fails in Google App Engine, this is the relevant part of the traceback:
File "/<>/test/main.py", line 7, in get
self.response.write(env.get_template('test.html').render())
File "/<>/google_appengine/lib/jinja2-2.6/jinja2/environment.py", line 719, in get_template
return self._load_template(name, self.make_globals(globals))
File "/<>/google_appengine/lib/jinja2-2.6/jinja2/environment.py", line 693, in _load_template
template = self.loader.load(self, name, globals)
File "/<>/google_appengine/lib/jinja2-2.6/jinja2/loaders.py", line 115, in load
source, filename, uptodate = self.get_source(environment, name)
File "/<>/google_appengine/lib/jinja2-2.6/jinja2/loaders.py", line 226, in get_source
if not self.provider.has_resource(p):
File "/<>/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 1170, in has_resource
return self._has(self._fn(self.module_path, resource_name))
File "/<>/google_appengine/lib/setuptools-0.6c11/pkg_resources.py", line 1218, in _has
"Can't perform this operation for unregistered loader type"
NotImplementedError: Can't perform this operation for unregistered loader type
When I use a FileSystemLoader instead it works, however this won't work in a zip I guess:
import common.templates
env = Environment(loader=FileSystemLoader(common.templates.__path__))
I have seen this SO question but this isn't the problem.
Update:
I just tested the example in the 1.7.6 SDK and this works, however I'd love to see this working on a older SDK (or the old_dev_appserver).

The problem, I think, was with pkg_resources.
Some of the os related functionality isn't implemented in GAE but either jinja2 or Google provided a workaround it seems.
These links illustrate the issue:
http://code.google.com/p/googleappengine/issues/detail?id=60
https://github.com/mitsuhiko/jinja2/issues/143
Jinja2 PackageLoader on google app engine
https://bitbucket.org/tarek/distribute/issue/73/pkg_resources-fails-on-google-app-engine
I've used
jinja_environment = jinja2.Environment(autoescape=True,
loader=jinja2.FileSystemLoader(os.path.join(os.path.dirname(__file__), 'templates')))
as boilerplate to get the templates from the "template" directory in the same directory as the file that is setting the definition. But the question is about files zipped as eggs and using PackageLoader
Is there an __init__.py file inside the package? You said it worked on 1.7.6, though.

Related

RuntimeError: Working outside of application context Error in Flask Server

I am trying to use the flask server, but since I faced an error, I started debugging it and by removing many codes to simplify the code and find finally reached to the following error::
Traceback (most recent call last):
File "C:\Code\SportsPersonClassifier\server\server.py", line 18, in <module>
print(classify_image())
File "C:\Code\SportsPersonClassifier\server\server.py", line 10, in classify_image
response = jsonify(util.classify_image(util.get_b64_for_virat()))
File "E:\Users\Acer\anaconda3\lib\site-packages\flask\json\__init__.py", line 358, in jsonify
if current_app.config["JSONIFY_PRETTYPRINT_REGULAR"] or current_app.debug:
File "E:\Users\Acer\anaconda3\lib\site-packages\werkzeug\local.py", line 436, in __get__
obj = instance._get_current_object()
File "E:\Users\Acer\anaconda3\lib\site-packages\werkzeug\local.py", line 565, in _get_current_object
return self.__local() # type: ignore
File "E:\Users\Acer\anaconda3\lib\site-packages\flask\globals.py", line 52, in _find_app
raise RuntimeError(_app_ctx_err_msg)
RuntimeError: Working outside of application context.
This typically means that you attempted to use functionality that needed
to interface with the current application object in some way. To solve
this, set up an application context with app.app_context(). See the
documentation for more information.
This is my code:: in server.py
from flask import Flask, request, jsonify
import util
app = Flask(__name__)
#app.route('/classify_image', methods=['GET', 'POST'])
def classify_image():
response = jsonify(util.classify_image(util.get_b64_for_image()))
return response
if __name__ == "__main__":
print("Starting Python Flask Server For Sports Celebrity Image Classification")
util.load_saved_artifacts()
print(classify_image())
But the exact code works without any error, if I just remove jsonify() from the classify_image() function like this::
def classify_image():
response = util.classify_image(util.get_b64_for_image())
return response
If I write the classify_image function without jsonify it works as expected without error. I tried to solve the problem reading several StackOverflow answers but not working for my code. Please help me solve the problem with jsonify. Thank you.
As the error suggests, and as required by jsonify:
This requires an active request or application context
Calling the function this way via __main__ will not be within a Flask app context. Instead use app.run() to start the dev server, and then navigate to your route.

Can't deploy python script to google cloud functions due to issue with flairnlp import

I am trying to deploy a Google Cloud Function that performs sentiment analysis on tweets using a flair nlp model. The code deploys perfectly fine without the line 'import flair' or alternatives like 'from flair import x,y,z'. As soon as I include the import statement for flair the function fails to deploy. Below is the error I get when deploying with the import statement (error is copied from Firebase logs). This is my first time posting on StackOverflow so please pardon me if the post looks ugly.
{"#type":"type.googleapis.com/google.cloud.audit.AuditLog","status":{"code":3,"message":"Function failed on loading user code. This is likely due to a bug in the user code. Error message: Code in file main.py can't be loaded.\nDetailed stack trace:\nTraceback (most recent call last):\n File \"/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py\", line 359, in check_or_load_user_function\n _function_handler.load_user_function()\n File \"/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v2.py\", line 236, in load_user_function\n spec.loader.exec_module(main_module)\n File \"<frozen importlib._bootstrap_external>\", line 728, in exec_module\n File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n File \"/user_code/main.py\", line 5, in <module>\n from flair import models, data\n File \"/env/local/lib/python3.7/site-packages/flair/__init__.py\", line 20, in <module>\n from . import models\n File \"/env/local/lib/python3.7/site-packages/flair/models/__init__.py\", line 1, in <module>\n from .sequence_tagger_model import SequenceTagger, MultiTagger\n File \"/env/local/lib/python3.7/site-packages/flair/models/sequence_tagger_model.py\", line 21, in <module>\n from flair.embeddings import TokenEmbeddings, StackedEmbeddings, Embeddings\n File \"/env/local/lib/python3.7/site-packages/flair/embeddings/__init__.py\", line 6, in <module>\n from .token import TokenEmbeddings\n File \"/env/local/lib/python3.7/site-packages/flair/embeddings/token.py\", line 10, in <module>\n from transformers import AutoTokenizer, AutoConfig, AutoModel, CONFIG_MAPPING, PreTrainedTokenizer\nImportError: cannot import name 'AutoModel' from 'transformers' (unknown location)\n. Please visit https://cloud.google.com/functions/docs/troubleshooting for in-depth troubleshooting documentation."},"authenticationInfo":
And here is the script I am trying to deploy, as well as the requirements.txt file
main.py
import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore
from datetime import datetime, timedelta
import flair
# or from flair import models, data
class FirestoreHandler():
cred = credentials.Certificate("serviceAccountKey.json")
firebase_admin.initialize_app(cred)
db = firestore.client()
def analysis_on_create(self):
docs = self.db.collection('tweets').order_by(u'time', direction=firestore.Query.DESCENDING).limit(1).get()
data = docs[0].to_dict()
most_recent_tweet = data['full-text']
sentiment_model = flair.models.TextClassifier.load('en-sentiment')
sentence = flair.data.Sentence(str(most_recent_tweet))
sentiment_model.predict(sentence)
result = sentence.labels[0]
if result.value == "POSITIVE":
val= 1 * result.score
else:
val= -1 * result.score
self.db.collection('sentiment').add({'sentiment':val,'timestamp':datetime.now()+timedelta(hours=3)})
def add_test(self):
self.db.collection('test3').add({"status":"success", 'timestamp':datetime.now()+timedelta(hours=3)})
def hello_firestore(event, context):
"""Triggered by a change to a Firestore document.
Args:
event (dict): Event payload.
context (google.cloud.functions.Context): Metadata for the event.
"""
resource_string = context.resource
# print out the resource string that triggered the function
print(f"Function triggered by change to: {resource_string}.")
# now print out the entire event object
print(str(event))
fire = FirestoreHandler()
fire.add_test()
fire.analysis_on_create()
requirements.txt
# Function dependencies, for example:
# package>=version
firebase-admin==5.0.1
https://download.pytorch.org/whl/cpu/torch-1.0.1.post2-cp37-cp37m-linux_x86_64.whl
flair
I included the url to pytorch download because flair is built on pytorch, and the function would not deploy without the url (even when I didn't import flair in main.py). I have also tried specifying different versions for flair to no avail.
Any intuition as to what may be causing this issue would be greatly appreciated! I am new to the Google Cloud ecosystem, this being my first project. If there is any additional information I can provide please let me know.
Edit: I am deploying from the website (not using CLI)
I am not sure that the provided requirements.txt is OK for GCP cloud functions deployment. Not sure that an explicit https URL is going to be handled correctly...
The Specifying dependencies in Python documentation page describes how the dependencies are to be stated - using the pip package manager's requirements.txt file or packaging local dependencies alongside your function.
Can you simply mention flair with necessary version in the requirements.txt file? Will it work?
In addition, the error you provided highlights that the transformers package is required. Can it be that some specific version is required?
====
As a side comment - I don't know your context and requirements, but I am not sure that in order to work with the Firestore from inside a cloud function all of that
import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore
is required, as it might be better to avoid using serviceAccountKey.json at all, and simply assign relevant IAM roles to the service account which is used for the given cloud function execution.

getting error 'function' object has no attribute 'as_view' while trying to run flask app

I started writing flask app after a long time more than a year, guess I have forgot something. This below code results in an error:
from flask import Flask
from flask import jsonify
from flask_restplus import Resource, Api
from home_iot.config import reader
from download_audio.ydla import download
app = Flask(__name__)
_api = Api(app, catch_all_404s=True, version=0.1,
title="REST HTTP API's Gateway",
descrition="REST API gateway")
api_ns = _api.namespace("iot", description="API.")
#api_ns.route("/tcpserver", methods=["GET"])
def advertise_tcpserver():
ip = reader.get_server_ip()
return jsonify({"tcpserver": ip})
if __name__ == "__main__":
app.run(host='127.0.0.1')
Error is:
$ python app.py
Traceback (most recent call last):
File "app.py", line 29, in <module>
#api_ns.route("/tcpserver", methods=["GET"])
File "/Users/ciasto/pyenvs/flaskrestplusiot/lib/python2.7/site-packages/flask_restplus/namespace.py", line 98, in wrapper
self.add_resource(cls, *urls, **kwargs)
File "/Users/ciasto/pyenvs/flaskrestplusiot/lib/python2.7/site-packages/flask_restplus/namespace.py", line 87, in add_resource
api.register_resource(self, resource, *ns_urls, **kwargs)
File "/Users/ciasto/pyenvs/flaskrestplusiot/lib/python2.7/site-packages/flask_restplus/api.py", line 264, in register_resource
self._register_view(self.app, resource, namespace, *urls, **kwargs)
File "/Users/ciasto/pyenvs/flaskrestplusiot/lib/python2.7/site-packages/flask_restplus/api.py", line 287, in _register_view
resource_func = self.output(resource.as_view(endpoint, self, *resource_class_args,
AttributeError: 'function' object has no attribute 'as_view'
Hope this can helps those who have this same error and have not found the solution
To complete the answer given by #v25 you must provide ressources to your namespace by inheriting from Ressource class in flask_restplus.
The following example works for me
Environment:
ubuntu 18.04
python 3.7.1
python requirements:
flask==1.1.2
flask-restplus==0.13.0
werkzeug==0.16.1
Source code:
iot.py
from flask_restplus import Namespace,Resource
api_ns = Namespace("iot", description="API.")
#api_ns.route("/tcpserver")
class AdvertiseTcpserver(Resource):
def get(self):
#TODO return the correct ip value
return {"tcpserver": "ip"}
app.py
from .iot import api_ns
from flask import Flask
from flask_restplus import Api
app = Flask(__name__)
_api = Api(app, catch_all_404s=True, version=0.1,
title="REST HTTP API's Gateway",
descrition="REST API gateway")
_api.add_namespace(api_ns, path='/some/prefix')
app.run()
Test command:
#!/bin/sh
wget localhost:5000/some/prefix/tcpserver
Please let me know if this helped.
Don't think that's the correct way to define the namespace with flask_restplus. Have a look at scaling docs.
You're probably looking for something like:
iot.py
from flask_restplus import Namespace
api_ns = Namespace("iot", description="API.")
#api_ns.route("/tcpserver", methods=["GET"])
def advertise_tcpserver():
ip = reader.get_server_ip()
return jsonify({"tcpserver": ip})
Then in your main app.py:
# other imports
from .iot import api_ns
app = Flask(__name__)
_api = Api(app, catch_all_404s=True, version=0.1,
title="REST HTTP API's Gateway",
descrition="REST API gateway")
_api.add_namespace(api_ns, path='/some/prefix')
Also you appear to be using Python 2.7 which has been discontinued. I'd suggest upgrading to the latest version, using either a virutal environment or docker so as not to mess with your system's python.

Pyramid fails to start when webtest and sqlalchemy are used together

I am trying to setup a pyramid app to use both webtest and sqlalchemy.
If I comment out the SQLAlchemy code, the webtests run without a problem.
[Test log ] https://travis-ci.org/caffeinated-expert/frisbee/builds/91622436
Error
Traceback (most recent call last):
File "frisbee/frisbee/tests/test_cities_page.py", line 18, in setUp
app = main({})
File "frisbee/frisbee/__init__.py", line 15, in main
engine = engine_from_config(settings, 'sqlalchemy.')
File "build/bdist.macosx-10.10-x86_64/egg/sqlalchemy/engine/__init__.py", line 426, in engine_from_config
url = options.pop('url')
KeyError: 'url'
In my main init file, if I comment out the sqlalchemy engine setup, then the tests run fine, but I need sqlalchemy for my project.
This is is the first time I have used webtest, so not sure if I have some other conflicting config.
[Failing code] https://github.com/caffeinated-expert/frisbee/commit/ea759015de755aca1d7bffca2845b72944572bed
From the sqlaclhemy docs:
The only required key is (assuming the default prefix) sqlalchemy.url
In your test_cities_page.py file you call main with an empty dictionary, presumably to be unpacked for **settings. You need to add the appropriate setting to the dictionary you're passing to the function and it should run. : )

Google App Engine with local Django 1.1 gets Intermittent Failures

I'm using the Windows Launcher development environment for Google App Engine.
I have downloaded Django 1.1.2 source, and un-tarrred the "django" subdirectory to live within my application directory (a peer of app.yaml)
At the top of each .py source file, I do this:
import settings
import os
os.environ["DJANGO_SETTINGS_MODULE"] = 'settings'
In my file settings.py (which lives at the root of the app directory, as well), I do this:
DEBUG = True
TEMPLATE_DIRS = ('html')
INSTALLED_APPS = ('filters')
import os
os.environ["DJANGO_SETTINGS_MODULE"] = 'settings'
from google.appengine.dist import use_library
use_library('django', '1.1')
from django.template import loader
Yes, this looks a bit like overkill, doesn't it?
I only use django.template. I don't explicitly use any other part of django.
However, intermittently I get one of two errors:
1) Django complains that DJANGO_SETTINGS_MODULE is not defined.
2) Django complains that common.html (a template I'm extending in other templates) doesn't exist.
95% of the time, these errors are not encountered, and they randomly just start happening. Once in that state, the local server seems "wedged" and re-booting it generally fixes it.
What's causing this to happen, and what can I do about it? How can I even debug it?
Here is the traceback from the error:
Traceback (most recent call last):
File "C:\code\kwbudget\edit_budget.py", line 34, in get
self.response.out.write(t.render(template.Context(values)))
File "C:\code\kwbudget\django\template\__init__.py", line 165, in render
return self.nodelist.render(context)
File "C:\code\kwbudget\django\template\__init__.py", line 784, in render
bits.append(self.render_node(node, context))
File "C:\code\kwbudget\django\template\__init__.py", line 797, in render_node
return node.render(context)
File "C:\code\kwbudget\django\template\loader_tags.py", line 71, in render
compiled_parent = self.get_parent(context)
File "C:\code\kwbudget\django\template\loader_tags.py", line 66, in get_parent
raise TemplateSyntaxError, "Template %r cannot be extended, because it doesn't exist" % parent
TemplateSyntaxError: Template u'common.html' cannot be extended, because it doesn't exist
And edit_budget.py starts with exactly the lines that I included up top.
All templates live in a directory named "html" in my root directory, and "html/common.html" exists. I know the template engine finds them, because I start out with "html/edit_budget.html" which extends common.html.
It looks as if the settings module somehow isn't applied (because that's what adds html to the search path for templates).
Firstly, although django is now a LOT more compatible with app engine than it once, some major incompatibilities still exist between the two platforms, meaning that you can't just dump a stock copy of django into your appengine directory and have it work out of the box. Things will error in strange ways.
There are a number of projects which aim to improve compatibility between the two projects, the most prominent is app-engine-patch. I highly suggest reading the following article http://code.google.com/appengine/articles/app-engine-patch.html and the rest of the articles located at code.google.com/appengine/articles/ under the django tab.
as for some of you're specific problems, you could try this within your setup script:
#setup django environment
from django.core.management import setup_environ
import settings
setup_envion(settings)
this is what django uses internally for setting up the environment within manage.py and is the accepted best practice for setting up django for use with scripts (like app engine).
I'm having exactly the same issue, and I haven't been able to work around it... though I've noticed it happens a LOT less with the real GAE than it does with the development server I run on my Linux workstation.

Categories