We Keep Coding

Python & GAE - StringType write() argument type parsing

When running dev_appserver.py ., I get the following error when trying to access http://localhost:8080:
Traceback (most recent call last):
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 86, in run
self.finish_response()
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
self.write(data)
File "/usr/local/Cellar/python#2/2.7.15_1/Frameworks/Python.framework/Versions 2.7/lib/python2.7/wsgiref/handlers.py", line 204, in write
assert type(data) is StringType,"write() argument must be string"
AssertionError: write() argument must be string
I've searched and it seems to come back to my app.yaml file, per these links:
SO Question for GAE Assertion Error
Russian Question site with
same information
I'm just not sure how to go about debugging it. Below is my app.yaml file and my main.py file. I'm super new to the GAE platform and any help would be appreciated.
app.yaml file:
application: gqtimer
version: 1-56
runtime: python27
api_version: 1
threadsafe: false
handlers:
- url: /favicon.ico
static_files: static/images/favicon.ico
upload: static/images/favicon.ico
- url: /_ah/login_required
script: main.py
- url: /static
static_dir: static
- url: /player.*
script: main.py
login: required
- url: /stat.*
script: main.py
login: required
- url: .*
script: main.py
libraries:
- name: django
version: "1.11"
main.py file:
#!/usr/bin/env python
#
import config
import os
import sys
# Force sys.path to have our own directory first, so we can import from it.
sys.path.insert(0, config.APP_ROOT_DIR)
sys.path.insert(1, os.path.join(config.APP_ROOT_DIR, 'externals'))
os.environ["DJANGO_SETTINGS_MODULE"] = "settings"
import wsgiref.handlers
from google.appengine.ext import webapp
from google.appengine.dist import use_library
use_library('django', '1.2')
from handlers import error, timer, do_openid_login
def main():
application = webapp.WSGIApplication([('/', timer.ExportHandler),
('/_ah/login_required', do_openid_login.OpenIdLoginHandler),
('/player/([-\w]+)', timer.PlayerHandler),
('/player/([-\w]+)/archives', timer.ArchivesHandler),
('/stat/([-\w]+)', timer.StatHandler),
('/stat/([-\w]+)/delete', timer.StatDeleteHandler),
# If we make it this far then the page we are looking
# for does not exist
('/.*', error.Error404Handler),
],
debug=True)
wsgiref.handlers.CGIHandler().run(application)
if __name__ == '__main__':
main()

Indeed, your app.yaml file might not be properly mapping your application code. You need to:
take the application variable outside of the main() function so that it becomes a global variable in the main.py module (and maybe rename it to app as well - just to stay inline with the official convention and documentation examples)
replace the script: main.py statements from your app.yaml's handlers with main.application (or main.app if renamed as mentioned above) - this is the reference to the above-mentioned global variable. From the script row in the Handlers element table:
A script: directive must be a python import path, for example,
package.module.app that points to a WSGI application. The last component of a script: directive using a Python module path is
the name of a global variable in the module: that variable must be a
WSGI app, and is usually called app by convention.
I'd also recommend explicitly passing the app.yaml as argument to dev_appserver.py instead of the app's directory (. in your case) - occasionally the auto-detection doesn't behave as expected. It's also the only way to run multiple services and/or using a dispatch.yaml file for routing, so it's a good habit.

How to import Django Settings to python standalone script

I'm having difficulty calling a django script. Here is what I'm currently doing in my root directory:
>>> import os
>>> os.environ['DJANGO_SETTINGS_MODULE'] = 'settings.py'
>>> from django.conf import settings
>>> settings.configure()
>>> settings.DATABASES
{}
The settings.DATABASES should not be empty, so I know I haven't initialized the project correct. How would I do this in django2.1? I used to be able to do this easily using import settings; setup_environ(settings), but not anymore.
Note: I'm looking to be able to run the above from any directory. Here is an example from trying to import my project from tmp:
(V) david$ cd /tmp && python
>>> import django
>>> from django.conf import settings
>>> settings.configure()
>>> django.setup()
>>> from users.models import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'users'

The 2.1 docs state that you need to use setup() now. Relevant snippet:
import django
from django.conf import settings
settings.configure()
django.setup()
# Now this script or any imported module can use any part of Django it needs.
from myapp import models
Here is the full documentation.

You don't need to run settings.configure(). To properly initialize, you could do something like:
if __name__ == '__main__':
import sys, os
sys.path.append(django_root)
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from django.conf import settings
print (settings.DATABASES)

I have had this problem and my solution (script) is the result of hundreds of Django sites over the course of a decade.
Some solutions can inhibit the use of settings and this is the best working version of a stand alone script for running Django that I have been able to compile. This is faster than the Django shell generally (no need to exit and re-enter).
Above suggestions will work 98% of the time. Still, consider reviewing (and commenting) on this for a more robust way to run Django scripts. I have hit the "Exceptions" about 1-2x every year for a while until I got this script fully debugged via many projects.
"""
scratch.py
----->note the no .py-----v
clear; echo 'import scratch' | python manage.py shell
"""
from django.conf import settings
from django.core.wsgi import get_wsgi_application # turns on app, access to DB
print(f'- - - - - - - - - - - - - - - - - - - - ')
print(f'settings: {settings}')
print('settings.DEBUG: {}'.format(settings.DEBUG))
# settings.DEBUG = True
# etc
if not settings.DEBUG:
# keeps this off prod, usually on git ignore as well
print('SETTINGS NOT IN DEBUG, set to:')
print(settings.DEBUG)
raise(RuntimeError('Can not be run on production or when debug is False'))
application = get_wsgi_application()
print('READY!')
print(f'- - - - - - - - - - - - - - - - - - - - ')
# App ready, code below. Add imports, calls, etc below here.
Additionally this script can be run in an infinite loop, such as for monitoring, debugging, and other day to day Django operations. At the ned of the file just add the following, possibly with a sleep() and logging.
while True:
pass
Call this script scratch.py from the same directory as manage.py. Call using the following syntax:
echo 'import scratch' | python manage.py shell
If you change the file name, then the import will need to be edited. This syntax loads the settings without needing to call settings.configure(). We leverage manage.py and that seems to solve the edge cases.
Note: This will NOT auto-reload on with changes.

mapping values are not allowed here ... in foo.py

I have this GAE python code
In file foo.py
import webapp2
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/plain'
self.response.write('Hello Foo')
app = webapp2.WSGIApplication([('/', MainPage)], debug = True)
in file app.yaml
application: foo
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: foo.app
I get this error pointing to the third line in file foo.py ( class MainPage(webapp2.RequestHandler): )
Obs. Begin reading from the end of the message
...
line 172, in _HandleEvents
for event in events:
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/yaml_listener.py", line 212, in _GenerateEventParameters
raise yaml_errors.EventListenerYAMLError(e)
google.appengine.api.yaml_errors.EventListenerYAMLError: mapping values are not allowed here
in "foo.py", line 3, column 39
I would appreciate a good help
thanks
Sam

This kind of error occurs if you start the application the wrong way: dev_appserver.py foo.py. You need a directory, e.g., foo with foo/foo.py and foo/app.yaml and then start the program from the parent directory with dev_appserver.py foo/ or in the directory itself with dev_appserver.py .

There is nothing wrong with your code. I copy and pasted both into files and ran them on my Win7 system using App Engine SDK release: "1.7.7" and it served up the page without errors.
You may have issues with the files or your setup.
Have you tried the File->Creating New Application menu option? It will create a new application called engineapp that will display "Hello world!" when browsed on the localhost machine.

Appstats are only working for one WSGIApplication

I've split handlers between 2 python files (main.py and main_cms.py). app.yaml defines the URLs that each python file will handle.
When I look at the Appstats, only the handlers from one of the 2 python files are profiled (the ones from main.py).
The 'magic' of webapp_add_wsgi_middleware(app) always used to work just fine, until the split. How can I make Appstats recording apply to all handlers?
appengine_config.py:
def webapp_add_wsgi_middleware(app):
from google.appengine.ext.appstats import recording
app = recording.appstats_wsgi_middleware(app)
return app
app.yaml:
builtins:
- appstats: on
...
- url: /services/.*
script: main.application
- url: /cms.*
script: main_cms.application
main.py and main_cms.py:
application = webapp2.WSGIApplication(url_mapping, config=config)
Running python2.7 / GAE 1.6.3, the dev_appserver.py shows Appstats correct for all handlers. The described problem is only seen in production.

What's wrong :lxml.html.parse in google app

Here is my app.yaml:
application: helloworld
version: 1
runtime: python27
api_version: 1
threadsafe: true
handlers:
- url: /.*
script: helloworld.app
libraries:
- name: lxml
version: latest
Here is helloworld.py:
from google.appengine.api import urlfetch
import lxml.html
down = "http://sc.hkex.com.hk/gb/www.hkex.com.hk/chi/market/sec_tradinfo/stockcode/eisdeqty_c.htm"
file = urlfetch.fetch(down)
print file.content
root = lxml.html.parse(file.content)
data = root.xpath('//tr[#class="tr_normal"]/td[position()=1]')
name = [row.text_content() for row in data]
for code in name:
print code
When I run:
google_appengine/dev_appserver.py helloworld/
I open 127.0.0.1:8080 and I can see the website http://sc.hkex.com.hk/gb/www.hkex.com.hk/chi/market/sec_tradinfo/stockcode/eisdeqty_c.htm but there is wrong output from the line print file.content.
May you tell me how to fix this?

lxml is a C module, GAE only supports pure python modules so it's not supported.