I've decided to try out this project:
http://code.google.com/p/appengine-admin/wiki/QuickStart
For the sake of the experiment, I took the demo guest-book shipped with App Engine. The import park look like this:
import cgi
import datetime
import wsgiref.handlers
from google.appengine.ext import db
from google.appengine.api import users
from google.appengine.ext import webapp
from google.appengine.ext.webapp.util import run_wsgi_app
from google import appengine_admin
The db model and the admin look like this:
class Greeting(db.Model):
author = db.UserProperty()
content = db.StringProperty(multiline=True)
date = db.DateTimeProperty(auto_now_add=True)
class AdminGreeting(appengine_admin.ModelAdmin):
model = Greeting
listFields = ('author','content','date')
editFields = ('author','content','date')
appengine_admin.register(AdminGreeting)
Yet I get this exception, trying to run the site:
File "/home/<username>/python/google_appengine/google/appengine/tools/ dev_appserver.py", line 2875, in _HandleRequest
base_env_dict=env_dict)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 387, in Dispatch
base_env_dict=base_env_dict)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 2162, in Dispatch
self._module_dict)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 2080, in ExecuteCGI
reset_modules = exec_script(handler_path, cgi_path, hook)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1976, in ExecuteOrImportScript
exec module_code in script_module.__dict__
File "/home/<username>/python/google_appengine/demos/guestbook/guestbook.py", line 37, in <module>
appengine_admin.register(AdminGreeting)
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 120, in register
modelAdminInstance = modelAdminClass()
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 64, in __init__
self._extractProperties(self.listFields, self._listProperties)
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 76, in _extractProperties
storage.append(PropertyWrapper(getattr(self.model, propertyName), propertyName))
File "/home/<username>/python/google_appengine/google/appengine_admin/model_register.py", line 17, in __init__
logging.info("Caching info about property '%s'" % name)
File "/usr/lib/python2.6/logging/__init__.py", line 1451, in info
root.info(*((msg,)+args), **kwargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1030, in info
self._log(INFO, msg, args, **kwargs)
File "/usr/lib/python2.6/logging/__init__.py", line 1142, in _log
record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, func, extra)
File "/usr/lib/python2.6/logging/__init__.py", line 1117, in makeRecord
rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
File "/usr/lib/python2.6/logging/__init__.py", line 272, in __init__
from multiprocessing import current_process
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1089, in decorate
return func(self, *args, **kwargs)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1736, in load_module
return self.FindAndLoadModule(submodule, fullname, search_path)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1089, in decorate
return func(self, *args, **kwargs)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1638, in FindAndLoadModule
description)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1089, in decorate
return func(self, *args, **kwargs)
File "/home/<username>/python/google_appengine/google/appengine/tools/dev_appserver.py", line 1589, in LoadModuleRestricted
description)
File "/usr/lib/python2.6/multiprocessing/__init__.py", line 83, in <module>
import _multiprocessing
ImportError: No module named _multiprocessing
INFO 2009-04-25 23:34:27,628 dev_appserver.py:2934] "GET / HTTP/1.1" 500 -
Any idea what could have went wrong?
You appear to be using Python 2.6 (given that some of the messages are from files in /usr/lib/python2.6 ...!), but Google App Engine needs Python 2.5 (any 2.5.x will do for any versions of x), so you should install and use that to run the App Engine SDK.
Google App Engine only supports Python 2.5, and you're on a newer version.
By the look of your directories, you might be on a Linux (or is it a Mac?). On, say, Ubuntu you can "sudo apt-get install python2.5" (it won't affect your Python 2.6 at all), and then rather than:
<path-to-gae>/dev_appserver.py ...
do
python2.5 <path-to-gae>/dev_appserver.py ...
This is better than just blithely developing on 2.6 and deploying on 2.5 which is surely asking for hassles later on.
As others have said, this problem occurs in Python 2.6. I used the fix proposed in this comment in the App Engine issue tracker:
A quickfix is to create a file in your app's root named `_multiprocessing.py' with
the contents:
import multiprocessing
This way it's possible to import the _multiprocessing module.
It worked for me using Python 2.6.2
Cheers,
Kaji
To make this work on my local machine (with 2.6) and on GAE, I used:
import sys, logging
if sys.version[:3] == "2.6": logging.logMultiprocessing = 0
just do the following at the top of your something.py
import logging
logging.logMultiprocessing = 0
import logging
logging.logMultiprocessing = 0
Worked for me
It wired that I have been using GAE with python2.6(probably 2.6.1), and every thing worked fine.
But now I get the same _multiprocess import error.(python2.6.2).
import logging
logging.logMultiprocessing = 0
Worked for me too. Before uploading to GAE, comment those lines.
Related
I want to setup a machine learning pipeline that is callable by flask but I am facing some issues, these links are for the documentations I have read so far:
https://exploreflask.com/en/latest/views.html#view-decorators
https://flask.palletsprojects.com/en/1.1.x/api/#flask.Flask
Let me explain the pipeline I have in mind:
pull a dataframe from a PostgreSQL database
encode said dataframe to make it ready for most algorithms
split up the data
feed to a pipeline and determine accuracy
store the model in a pickle file
What is working so far:
All parts are working as a regular script
I can just slap all the steps into one huge flask file with one decorator and it would run as well (my emergency solution)
The File Structure
The encoder script:
#Flask main thread
#makes flask start this part as application and not as module
app = Flask('encoder_module')
#app.route('/df_encoder')
def df_encoder(rng = 4):
encoding stuff
`return df`
The Pipeline script (random forest regressor here)
app = Flask('pipeline_module')
#app.route('/pipeline_rfr')
def pipeline_rfr():
pipeline stuff
`return grid_search_rfr`
The pickle module:
app = Flask('pickle_module')
#app.route('/store_reg_pickle')
def store_pickle():
"""
Usage of a Pickle Model -Storage of a trained Model
"""
model = grid_search_rfr
#specify file name in letter strings
model_file = "regression_model"
with open(model_file, mode='wb') as m_f:
pickle.dump(model, m_f)
print(f"Model saved in: {os.getcwd()}")
return model_file
The Main Flask File
#packages
from flask import Flask
from encoder_main_thread import df_encoder
from rfr_pipeline_function import pipeline_rfr
from pickle_call import store_pickle
app = Flask(__name__.split('.')[0])
#app.route('/regression_pipe')
#df_encoder
#pipeline_rfr
#store_reg_pickle
def regression_pipe():
`return 'pipeline done`
The problem iss that the return value of the encoder cannot be a dataframe, only a string, tuple, etc.
Is there a workaround for this?
I actually want it to be a flawless passing of the dataframe to the pipeline and eventually storing it in the pickle file which is then saved in the folder.
For some reason it cannot detect the pickle file import and throws following error:
Use a production WSGI server instead.
* Debug mode: off
Traceback (most recent call last):
File "C:\ANACONDA3\Scripts\flask-script.py", line 9, in <module>
sys.exit(main())
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 967, in main
cli.main(args=sys.argv[1:], prog_name="python -m flask" if as_module else None)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\click\decorators.py", line 73, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 848, in run_command
app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 305, in __init__
self._load_unlocked()
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 330, in _load_unlocked
self._app = rv = self.loader()
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 388, in load_app
app = locate_app(self, import_name, name)
File "C:\ANACONDA3\lib\site-packages\flask\cli.py", line 240, in locate_app
__import__(module_name)
File "C:\Users\bill-\OneDrive\Dokumente\Docs Bill\TA_files\functions_scripts_storage\flask_test\flask_regression_pipeline.py", line 18, in <module>
#store_reg_pickle
NameError: name 'store_reg_pickle' is not defined
If you wish I could upload the entire scripts but that is a lot to look through and since it is working as a long regular pice of code, the mistake needs to be somewhere with my flask setup.
every time i run the code i see Serving Flask-SocketIO app "app.py"
i don't even have SpcketIO on my system
import os, passlib ,requests ,time ,json
from flask import Flask, session , render_template , request,redirect,url_for ,jsonify
from flask_session import Session
import pandas as pd
from flask_uploads import UploadSet, configure_uploads, IMAGES, patch_request_class
from werkzeug import secure_filename
from datetime import date , datetime
from flask_sqlalchemy import SQLAlchemy
from passlib.hash import sha256_crypt
from flask_login import LoginManager , UserMixin , current_user ,login_user ,login_required ,logout_user
app = Flask(__name__)
by the end of my code I've this
if __name__ == "__main__":
app.run(debug=True)
i run the system by cmd: flask run
I used to ignore this error as everything else was working but when i tried to add app.run(debug=True,host='0.0.0.0')
also debug=True didn't seem to be doing anything anymore i had to turn it on through the CMD set Flask_Debug=1
i got an error stating ValueError: signal only works in main thread
and the app didn't run at all
the exact error
* Serving Flask-SocketIO app "app.py"
* Forcing debug mode on
* Restarting with stat
* Debugger is active!
* Debugger PIN: 885-769-473
Exception in thread Thread-1:
Traceback (most recent call last):
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\threading.py
", line 917, in _bootstrap_inner
self.run()
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\threading.py
", line 865, in run
self._target(*self._args, **self._kwargs)
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\site-package
s\flask_socketio\cli.py", line 59, in run_server
return run_command()
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\site-package
s\click\core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\site-package
s\click\core.py", line 717, in main
rv = self.invoke(ctx)
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\site-package
s\click\core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\site-package
s\click\core.py", line 555, in invoke
return callback(*args, **kwargs)
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\site-package
s\click\decorators.py", line 64, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\site-package
s\click\core.py", line 555, in invoke
return callback(*args, **kwargs)
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\site-package
s\flask\cli.py", line 771, in run_command
threaded=with_threads, ssl_context=cert)
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\site-package
s\werkzeug\serving.py", line 812, in run_simple
reloader_type)
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\site-package
s\werkzeug\_reloader.py", line 267, in run_with_reloader
signal.signal(signal.SIGTERM, lambda *args: sys.exit(0))
File "c:\users\mena\appdata\local\programs\python\python37-32\lib\signal.py",
line 47, in signal
handler = _signal.signal(_enum_to_int(signalnum), _enum_to_int(handler))
ValueError: signal only works in main thread
to get back to work i had to turn off the debug in cmd set Flask_Debug=0
what's wrong with my system ?
please check if there is any other thread running or not. And try running your app by command python app.py from terminal instead of flask run. may be flask run command invoking other services to which are not necessary to your application.
Also I am assuming you are working on Linux system. In terminal (Windows cmd) and in your project directory where your app.py is located just enter python (python/python3 depnds whichever version you are using) app.py and hit enter.
try this on command line
python app.py
or
python3 app.py
If I try to run a test via PyCharm, I get this exception
...bin/python /usr/local/pycharm-4.5.1/helpers/pycharm/utrunner.py ...src/foo/foo/tests/FooEditTest.py::FooEditTest::test_issue_add true
Testing started at 15:59 ...
Traceback (most recent call last):
File "/usr/local/pycharm-4.5.1/helpers/pycharm/utrunner.py", line 139, in <module>
module = loadSource(a[0])
File "/usr/local/pycharm-4.5.1/helpers/pycharm/utrunner.py", line 41, in loadSource
module = imp.load_source(moduleName, fileName)
File "...src/foo/foo/tests/FooEditTest.py", line 45, in <module>
from foo.views.issue.forward import forward
File "...src/foo/foo/views/issue/forward.py", line 29, in <module>
class ForwardForm(forms.Form):
File "...src/foo/foo/views/issue/forward.py", line 36, in ForwardForm
group=forms.ModelChoiceField(Group.objects.exclude(groupconfig__no_issue=True).extra(
File "...python2.7/site-packages/django/db/models/manager.py", line 92, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "...python2.7/site-packages/django/db/models/query.py", line 698, in exclude
return self._filter_or_exclude(True, *args, **kwargs)
File "...python2.7/site-packages/django/db/models/query.py", line 707, in _filter_or_exclude
clone.query.add_q(~Q(*args, **kwargs))
File "...python2.7/site-packages/django/db/models/sql/query.py", line 1331, in add_q
clause, require_inner = self._add_q(where_part, self.used_aliases)
File "...python2.7/site-packages/django/db/models/sql/query.py", line 1358, in _add_q
current_negated=current_negated, connector=connector)
File "...python2.7/site-packages/django/db/models/sql/query.py", line 1182, in build_filter
lookups, parts, reffed_aggregate = self.solve_lookup_type(arg)
File "...python2.7/site-packages/django/db/models/sql/query.py", line 1120, in solve_lookup_type
_, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
File "...python2.7/site-packages/django/db/models/sql/query.py", line 1383, in names_to_path
field, model, direct, m2m = opts.get_field_by_name(name)
File "...python2.7/site-packages/django/db/models/options.py", line 416, in get_field_by_name
cache = self.init_name_map()
File "...python2.7/site-packages/django/db/models/options.py", line 445, in init_name_map
for f, model in self.get_all_related_m2m_objects_with_model():
File "...python2.7/site-packages/django/db/models/options.py", line 563, in get_all_related_m2m_objects_with_model
cache = self._fill_related_many_to_many_cache()
File "...python2.7/site-packages/django/db/models/options.py", line 577, in _fill_related_many_to_many_cache
for klass in self.apps.get_models():
File "...python2.7/site-packages/django/utils/lru_cache.py", line 101, in wrapper
result = user_function(*args, **kwds)
File "...python2.7/site-packages/django/apps/registry.py", line 168, in get_models
self.check_models_ready()
File "...python2.7/site-packages/django/apps/registry.py", line 131, in check_models_ready
raise AppRegistryNotReady("Models aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Models aren't loaded yet.
Process finished with exit code 1
I use group=forms.ModelChoiceField(Group.objects.exclude(...))
This line gets executed during importing. The call to django.setup() was not done before.
I have no clue how to solve this:
Should I call django.setup()? But where to insert this line?
Avoid using MyModel.objects.filter(...) during import time? This would need a big refactoring, since we have several ModelChoiceFields.
You're currently running the tests using the PyCharm's Python unit test runner pycharm-4.5.1/helpers/pycharm/utrunner.py.
This test runner is great for low-level unit tests (subclassed from unittest.TestCase) that don't touch Django features like the ORM, but
if the tests rely on Django-specific things like the database, then your TestCases probably need to be subclassed from django.test.testcases.TestCase and you'll need to go through PyCharm's Django test manager django_test_manage.py, which takes care of setting up the test database, initialising the model registry, and so on.
Run > Edit Configurations > Add New Configuration (+ button) > Django Tests
Set the target to foo.foo.tests.FooEditTest and make sure to put DJANGO_SETTINGS_MODULE=... in the environment variables if it doesn't find your settings.
I faced the same problem while running test cases and solved it by adding this to test file at the beginning with import statements
import os
import sys
from django.core.handlers.wsgi import WSGIHandler
os.environ['DJANGO_SETTINGS_MODULE'] = 'myapp.settings'
application = WSGIHandler()
Please show your exact code (especially the extra() clause). Calling filter() or exclude() at import time is not bad because querysets are lazy but you could evaluate the queryset here that caused the exception.
Do not evaluate querysets during import time because import statements are executed only once: e.g., if a new group is created, it won't be available as a choice for you ModelChoiceField.
A bit hard to say without seeing your code, but I had a similar issue once. For me it was related to a module that was being imported by my models and that also contained an import of my models.
ModelA --- imports --> service -- imports --> ModelA
I solved this by moving the import in my service to the method that needed the import. So instead of putting it at the top of the service module, I limited the scope of the import to the method that needed this import, thus avoiding importing the models whilst initializing the service module. Fwew, I wish I could draw it for you :)
I have seen some questions about using SQLAlchemy on App Engine to connect to Google Cloud SQL. But I'm not sure if it is possible to develop using a local MySQL database and the existing SQLAlchemy dialect. On my first attempt, I added SQLAlchemy 0.8.0 to the app and defined a schema:
from sqlalchemy import create_engine, Column, Integer, Table
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
foo_table = Table('foo', Base.metadata,
Column('id', Integer, primary_key=True, autoincrement=True),
)
And when I tried to create the tables on the development server using:
url = 'mysql+gaerdbms:///%s?instance=%s' % ('database_name', 'instance_name')
engine = create_engine(url)
Base.metadata.create_all(engine)
...I got an error DBAPIError: (ImportError) No module named pwd None None, which means that SQLAlchemy is importing a module that is blacklisted by the development server.
Am I doing something wrong? Or, if not, what should I do to use SQLAlchemy on the development server? Or maybe the first question is: Can I use the SQLAlchemy's gaerdbms dialect to develop in a local MySql database using the dev server?
Edit: this error doesn't happen only when trying to create tables. I created the tables manually and tried to query them, and the same error occurs.
The full traceback is:
Traceback (most recent call last):
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "[...]/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "[...]/webapp/admin.py", line 12, in get
db.Base.metadata.create_all(engine)
File "[...]/webapp/sqlalchemy/schema.py", line 2784, in create_all
tables=tables)
File "[...]/webapp/sqlalchemy/engine/base.py", line 1486, in _run_visitor
with self._optional_conn_ctx_manager(connection) as conn:
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "[...]/webapp/sqlalchemy/engine/base.py", line 1479, in _optional_conn_ctx_manager
with self.contextual_connect() as conn:
File "[...]/webapp/sqlalchemy/engine/base.py", line 1669, in contextual_connect
self.pool.connect(),
File "[...]/webapp/sqlalchemy/pool.py", line 272, in connect
return _ConnectionFairy(self).checkout()
File "[...]/webapp/sqlalchemy/pool.py", line 425, in __init__
rec = self._connection_record = pool._do_get()
File "[...]/webapp/sqlalchemy/pool.py", line 855, in _do_get
return self._create_connection()
File "[...]/webapp/sqlalchemy/pool.py", line 225, in _create_connection
return _ConnectionRecord(self)
File "[...]/webapp/sqlalchemy/pool.py", line 318, in __init__
self.connection = self.__connect()
File "[...]/webapp/sqlalchemy/pool.py", line 368, in __connect
connection = self.__pool._creator()
File "[...]/webapp/sqlalchemy/engine/strategies.py", line 80, in connect
return dialect.connect(*cargs, **cparams)
File "[...]/webapp/sqlalchemy/engine/default.py", line 279, in connect
return self.dbapi.connect(*cargs, **cparams)
File "[...]/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 183, in __init__
super(GoogleApiConnection, self).__init__(*args, **kwargs)
File "[...]/google_appengine/google/storage/speckle/python/api/rdbms.py", line 810, in __init__
self.OpenConnection()
File "[...]/google_appengine/google/storage/speckle/python/api/rdbms.py", line 832, in OpenConnection
self.SetupClient()
File "[...]/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 193, in SetupClient
self._client = RdbmsGoogleApiClient(**kwargs)
File "[...]/google_appengine/google/storage/speckle/python/api/rdbms_googleapi.py", line 106, in __init__
rdbms.OAUTH_CREDENTIALS_PATH)
File "/usr/lib/python2.7/posixpath.py", line 259, in expanduser
import pwd
File "[...]/google_appengine/google/appengine/tools/devappserver2/python/sandbox.py", line 822, in load_module
raise ImportError('No module named %s' % fullname)
DBAPIError: (ImportError) No module named pwd None None
I found a workaround. As it is, SQLAlchemy's gaerdbms dialect can't connect to a local database. But with the dialect below it can. Folow the instructions from this answer but use this dialect instead:
# mysql/gaerdbms.py
# Copyright (C) 2005-2013 the SQLAlchemy authors and contributors <see AUTHORS file>
#
# This module is part of SQLAlchemy and is released under
# the MIT License: http://www.opensource.org/licenses/mit-license.php
"""
.. dialect:: mysql+gaerdbms
:name: Google Cloud SQL
:dbapi: rdbms
:connectstring: mysql+gaerdbms:///<dbname>?instance=<instancename>
:url: https://developers.google.com/appengine/docs/python/cloud-sql/developers-guide
This dialect is based primarily on the :mod:`.mysql.mysqldb` dialect with minimal
changes.
.. versionadded:: 0.7.8
Pooling
-------
Google App Engine connections appear to be randomly recycled,
so the dialect does not pool connections. The :class:`.NullPool`
implementation is installed within the :class:`.Engine` by
default.
"""
import os
import re
from sqlalchemy.dialects.mysql.mysqldb import MySQLDialect_mysqldb
from sqlalchemy.pool import NullPool
class MySQLDialect_gaerdbms(MySQLDialect_mysqldb):
#classmethod
def dbapi(cls):
# from django:
# http://code.google.com/p/googleappengine/source/
# browse/trunk/python/google/storage/speckle/
# python/django/backend/base.py#118
# see also [ticket:2649]
# see also https://stackoverflow.com/q/14224679/34549
if is_production():
# Production mode.
from google.storage.speckle.python.api import rdbms_apiproxy
return rdbms_apiproxy
elif is_remote_mode():
# Development mode with remote database.
from google.storage.speckle.python.api import rdbms_googleapi
return rdbms_googleapi
else:
# Development mode with local database.
from google.appengine.api import rdbms_mysqldb
return rdbms_mysqldb
#classmethod
def get_pool_class(cls, url):
# Cloud SQL connections die at any moment
return NullPool
def create_connect_args(self, url):
opts = url.translate_connect_args()
if is_production() or is_remote_mode():
# 'dsn' and 'instance' are because we are skipping
# the traditional google.api.rdbms wrapper.
# they are not needed in local mode; 'dns' even causes an error.
opts['dsn'] = ''
opts['instance'] = url.query['instance']
return [], opts
def _extract_error_code(self, exception):
match = re.compile(r"^(\d+):|^\((\d+),").match(str(exception))
# The rdbms api will wrap then re-raise some types of errors
# making this regex return no matches.
code = match.group(1) or match.group(2) if match else None
if code:
return int(code)
dialect = MySQLDialect_gaerdbms
def is_production():
return os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine')
def is_remote_mode():
return os.getenv('SETTINGS_MODE') == 'prod'
This dialect uses a local database by default when running on the development server. To use remote access to Google Cloud SQL during development, a variable must be set in the environment, following the pattern used by Django:
os.environ['SETTINGS_MODE'] = 'prod'
I have this piece of code which is running perfectly on localhost but throws up this obscure error on GAE:
import_string() failed for 'webapp2_extras.appengine.auth.models.User' . Possible reasons are: - missing __init__.py in a package; - package or module
My import statements:
from webapp2_extras import auth
from webapp2_extras import sessions
from webapp2_extras.auth import InvalidAuthIdError
from webapp2_extras.auth import InvalidPasswordError
Usage of auth's user model:
user = self.auth.store.user_model.create_user(username, password_raw = password, email = email)
if not user[0]: #returns a tuple with [boolean, user_info]
return 'Create user error'
else:
self.set_flash("Thank you for registering. Please login!")
self.redirect(self.auth_config['login_url'])
Full code
UPDATE (Full stack trace)
import_string() failed for 'webapp2_extras.appengine.auth.models.User'. Possible reasons are:
- missing __init__.py in a package;
- package or module path not included in sys.path;
- duplicated package or module name taking precedence in sys.path;
- missing module, class, function or variable;
Original exception:
ImportError: No module named ndb
Debugged import:
- 'webapp2_extras' found in '/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2_extras/__init__.pyc'.
- 'webapp2_extras.appengine' found in '/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2_extras/appengine/__init__.pyc'.
- 'webapp2_extras.appengine.auth' found in '/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2_extras/appengine/auth/__init__.pyc'.
- 'webapp2_extras.appengine.auth.models' not found.
Traceback (most recent call last):
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1511, in __call__
rv = self.handle_exception(request, response, e)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1505, in __call__
rv = self.router.dispatch(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1253, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1077, in __call__
return handler.dispatch()
File "/base/data/home/apps/s~webapp-auth/1.358936463581927371/main.py", line 34, in dispatch
response = super(BaseHandler, self).dispatch()
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 547, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~webapp-auth/1.358936463581927371/main.py", line 127, in post
user = self.auth.store.user_model.create_user(username, password_raw = password, email = email)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 701, in __get__
value = self.func(obj)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2_extras/auth.py", line 131, in user_model
cls = self.config['user_model'] = webapp2.import_string(cls)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 1824, in import_string
return getattr(__import__(module, None, None, [obj]), obj)
File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2_extras/appengine/auth/models.py", line 13, in <module>
from ndb import model
Look similar to this issue which was already fixed by webapp 2.5.1
Make sure you import the latest version of webapp2, by adding those line to your app.yaml file:
libraries:
- name: webapp2
version: latest
As a workaround you can add the following lines to your application:
import sys
from google.appengine.ext import ndb
sys.modules['ndb'] = ndb