I'm trying to figure out how to force a flush of the GAE datastore to disk in the dev appserver.
I understand that some time ago writes were disabled until dev_appserver shutdown from here:
App Engine local datastore content does not persist
My app is running 24/7 doing things and generating data, and I do not want to have to restart things from zero if the VM hosting my dev_appserver goes down or something. So, I'd like to schedule a periodic flush of the datastore.
I found this method for 1.6.4:
GAE SDK 1.6.4 dev_appserver datastore flush
That doesn't work anymore, and I haven't found a variant using the API server that I am permitted to invoke from my app. Anyone know how to flush to disk on demand in 1.9?
No, dev_appserver is not designed to be used in production and provides no reliability or durability guarantees.
Ideally you'd be using the service on GCP, however if for some reason that is not possible you can look at the open source project AppScale that is designed for feature parity of Google App Engine / Cloud Datastore.
Related
This used to work until about 3 months ago.
dev_appserver.py --storage_path=/home/stephen/webprog/trunk2/ trunk2/app.yaml
now it starts the server but datastore is empty. I need the DataStore as it has models and user logins. What I do now is to unzip an old trunk from 3 months ago and start the server, then change the code (not touching the DS models), then log in, test, logout and stop with C-c. if I restrat the server the DataStore will not load. How can I make it load the DataStore so that I do not need to revert to my old working trunk? I am using Google App Engine using python.
Thanks
As specified in the Local Development Server docs:
We are migrating the local development environment to use the Cloud Datastore Emulator, For more information about this change, see the migration guide.
You should follow the migration guide to use the Datastore Emulator instead
How can I edit the Google App Engine NGINX configuration?
There doesn't seem to be much support in the Google docs in regards to the NGINX configuration for apps running in the Google App Engine flexible environment.
My app is running fine, but I get this 413 error when I try and upload an audio file (.wav or .mp3).
413 Request Entity Too Large -- nginx
My app is running Django (python 3), with Cloud Postgres SQL and Cloud Storage enabled.
I researched the error, and it seems I can set a nginx.config file so that it includes "client_max_body_size 80M" - but like I said, there is no documentation regarding how to manually config NGINX on deploy.
Any suggestions?
You should be able to create a nginx-app.conf file in the same directory as your app.yaml file. There is an example of using the nginx configuration file in a Flex environment located here: https://github.com/GoogleCloudPlatform/getting-started-php/tree/master/4-auth .
This same file is referenced in Google's documentation here: https://cloud.google.com/appengine/docs/flexible/php/runtime#customizing_nginx
Once you have that file created, you should be able to add any property you need and then rebuild your project to see the changes take effect.
So upon contacting Google Support, the suggested solution for uploading files larger than 32MB is as follows:
"The way to circumvent App Engine's 32MB limit is to send the requests directly to Cloud Storage, for instance using the resumable upload process. You can still use App Engine to serve your app, but the clientside portion of the app would be the one handling the upload to Google Cloud Storage. For this you would have your application generate a signed URL which the client can use to gain access to your Cloud Storage bucket for the purpose of uploading an image."
I went with this solution. It saves money in the end.
I'm developing a Flask app with a microservices architecture in Google App Engine's Standard Environment. This app will need to sustain bursts of intense traffic, so it seems perfect for the Standard Environment. My question is this:
In other environments I've used (Heroku, App Engine Flexible), you must configure your Flask app to use a production-quality WSGI web server like gunicorn, as the built-in Flask server is only suitable for development. There is documentation on this for the Flexible Environment, but not for Standard. Am I correct in assuming that this is because the Standard Environment (as configured in app.yaml) manages the request handling and everything else that gunicorn would in a production-ready way? Could it really be that easy?
Yes, it really is that easy. As a PaaS, GAE takes care of all that for you. As it did for SnapChat when it grew to 150 million users (yes, hosted on GAE).
Watch "App Engine Architecture and Services" and "You Can Run That On App Engine?". These describe some of this. Incoming requests are first captured in a queue on a GAE front-end servers. From there it decides what to do with it. If an instance of your app isn't running, it starts one and then passes the request to it. If an instance is running and not too busy, it passes the request to it immediately. If all running instances are busy, it will hold the request until an instance is able to take on another request. If the request sits in the front-end queue for too long (based on parameters you can set), GAE will start up more instances to handle the backlog.
With GAE, your app now runs in an environment with load balancers, front-end request queuing servers, edge servers, auto-scaling app servers, private global fibre networks, etc, etc. This means all the "production quality" worries that would lead you to gunicorn, etc on a self-designed server are more than handled by Google's smart engineers in GAE.
Since you're looking a microservices, you might also want to read Microservices Architecture on Google App Engine, in GAE docs.
I am in the process of migrating my python application from Google App Engine to Heroku. I have managed to get webapp2 (google app engine's default request handler) working.
What I am experiencing an issue with, is the database. In GAE, I was using the built in Datastore, and my application contains all the database syntax according to that. But after migrating to Heroku, I have a lot of database choices as "addons".
I wanted to know, as to which database I should choose in Heroku, which resembles the Google App engine's datastore the most, so that I have to change the least amount of code in my Application.
Just as a reference, here's the link showing the vast amount of options available on Heroku for database choices: Heroku Database Addons.
Something like Cassandra: https://addons.heroku.com/cassandraio would be closest in performance to the GAE datastore given it's column-structured nature.
However, depending on your usage patterns and how close you stuck to the Django-like API, any of the standard datastores (Postgres, ClearDB MySQL) would be about the same amount of effort to switch over to from a code perspective.
I see google provide SDK and utilties to develop and run the web application in development (developer-pc) and port them to google app engine live (at google server).
Can we use google app engine to run the local web application without using google infrastructure?
Basically I want a decent job scheduler and persistent job queue for python (I am not using google infrastructure). I see google provides task queue implementation along with their app engine sdk.
Can I use google app engine SDK to development my full fledged python application for task queue?
You can run App Engine apps on top of appscale which in turn does run on Eucalyptus, Xen, and other clustering solutions you can deploy on Ubuntu (not sure about there being any Windows support) -- looks like it may require substantial system installation, configuration, and administration work to get started (sorry, no first-hand experience yet), but once you've done that investment it appears it may be smoother going forwards. (Automation of task queues is a relatively recent addition to appscale, but it's apparently working and can be patched in from a bazaar branch until it gets fully integrated into the trunk of the appscale project).
Edit: since there seems to be some confusion about licensing of this code, I'll point out that the App Engine SDK, as per its site, is under Apache License 2.0, and appscale's under the New BSD License. Both are extremely permissive and liberal open-source licenses that basically allow you all sorts of reuses, remixes, mashups, redistributions, etc, etc.
Edit: Nick also suggests mentioning TwistedAE, another effort to build an open source way (also Apache License 2.0) to deploy App Engine apps on your own infrastructure; I have no direct experience with it, and it IS still pre-alpha, but it does seem very promising and well worth keeping an eye on (tx Nick!).
I don't believe so. According to the App Engine terms of service:
7.1. Google gives you a personal, worldwide, royalty-free,
non-assignable and non-exclusive
license to use the software provided
to you by Google as part of the
Service as provided to you by Google
(referred to as the "Google App Engine
Software" below). This license is for
the sole purpose of enabling you to
use and enjoy the benefit of the
Service as provided by Google, in the
manner permitted by the Terms.
(emphasis mine)
You'd want to check with a lawyer, but to me this sounds like the dev_appserver.py server is only to be used for development of applications which are then deployed to the GAE "service", not for running your own servers internally.
I also suspect that running a production service off dev_appserver.py would be inadvisable for performance reasons. Without special effort, threaded Python web servers can generally only accomodate one request at a time, which limits your performance and scalability. This is due to an implementation detail of CPython, called the GIL. See http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-lock for a detailed explanation.