I'm making a blog using Google App Engine, which for now just allows me to post entries and see them listed out. I was following the Web Development course by Steve Huffman on Udacity where he showed a simple caching technique to decrease the database queries. Here's my code (which he instructed):
CACHE = {}
def top_entries():
key = 'top'
# logging.error(CACHE)
if key in CACHE:
entries = CACHE[key]
else:
logging.error("DB QUERY")
entries = db.GqlQuery('select * from Entries order by created desc limit 10')
entries = list(entries)
CACHE[key] = entries
return entries
class MainPage(webapp2.RequestHandler):
def get(self):
entries = top_entries()
self.response.write(render_str('mainpage.html', entries=entries))
mainpage.html is just the main page that lists the 10 most recent entries.
Basically, CACHE is a dictionary that stores a key that identifies a particular database query (in this case, listing out the top 10 entries) and on further calls to the same query, the application simply looks in that dictionary without making a database call.
What SHOULD happen is this:
I load the page for the first time and DB QUERY is printed on my console since it's a database call. The result of the query is stored in CACHE.
I reload and DB QUERY isn't printed since the cache has the key.
The above DOES happen but only if I have a fresh new tab to work on. If I already have a tab where I was working on this app, and I shut down the server, reboot it and then try reloading the page again, the above doesn't work. What happens is that step 1 happens twice and then step 2. It seems like the key isn't stored in the CACHE the first time I reload the page, but on the second time.
Here's the console log if the above paragraph is hard to follow:
The first time I boot the server and start working on a fresh tab:
PS C:\Users\IBM_ADMIN> python `C:\Program Files (x86)\Google\google_appengine\dev_appserver.py' 'C:\Users\IBM_ADMIN\Downloads\Udacity\Web Development\Blog Project'
INFO 2016-06-16 14:31:04,000 sdk_update_checker.py:229] Checking for updates to the SDK.
INFO 2016-06-16 14:31:06,525 api_server.py:205] Starting API server at: http://localhost:53305
INFO 2016-06-16 14:31:06,545 dispatcher.py:197] Starting module "default" running at: http://localhost:8080
INFO 2016-06-16 14:31:06,552 admin_server.py:116] Starting admin server at: http://localhost:8000
ERROR 2016-06-16 09:01:15,358 blog.py:134] DB QUERY
INFO 2016-06-16 14:31:15,529 module.py:787] default: "GET / HTTP/1.1" 200 1849
INFO 2016-06-16 14:31:17,984 module.py:787] default: "GET / HTTP/1.1" 200 1849
INFO 2016-06-16 14:31:45,944 shutdown.py:45] Shutting down.
INFO 2016-06-16 14:31:46,040 api_server.py:648] Applying all pending transactions and saving the datastore
INFO 2016-06-16 14:31:46,042 api_server.py:651] Saving search indexes
And this is what happens when I don't open a fresh tab but start working on the old tab.
PS C:\Users\IBM_ADMIN> python `C:\Program Files (x86)\Google\google_appengine\dev_appserver.py' 'C:\Users\IBM_ADMIN\Downloads\Udacity\Web Development\Blog Project'
INFO 2016-06-16 14:31:56,470 sdk_update_checker.py:229] Checking for updates to the SDK.
INFO 2016-06-16 14:31:58,637 api_server.py:205] Starting API server at: http://localhost:53318
INFO 2016-06-16 14:31:58,651 dispatcher.py:197] Starting module "default" running at: http://localhost:8080
INFO 2016-06-16 14:31:58,657 admin_server.py:116] Starting admin server at: http://localhost:8000
ERROR 2016-06-16 09:02:08,336 blog.py:134] DB QUERY
INFO 2016-06-16 14:32:08,526 module.py:787] default: "GET / HTTP/1.1" 200 1849
ERROR 2016-06-16 09:02:12,538 blog.py:134] DB QUERY
INFO 2016-06-16 14:32:12,684 module.py:787] default: "GET / HTTP/1.1" 200 1849
INFO 2016-06-16 14:32:16,822 module.py:787] default: "GET / HTTP/1.1" 200 1849
INFO 2016-06-16 14:32:21,428 shutdown.py:45] Shutting down.
INFO 2016-06-16 14:32:21,430 api_server.py:648] Applying all pending transactions and saving the datastore
INFO 2016-06-16 14:32:21,430 api_server.py:651] Saving search indexes
DB QUERY gets printed out twice instead of once. I can't think of any reason why it would behave this way. (Maybe because of my browser cookies?)
You are caching at instance level and you can have number of instances or an instance can be recycled (shut down) and a new one instantiated on a request.
Try to use Memcache if you want to share cache between instances:
https://cloud.google.com/appengine/docs/python/memcache/
You can combine both approaches (instance cache + memcache)
Related
I'm working in PyCharm and building endpoints with Flask, and I use Pyctuator as well. It is connected to a Spring Boot Admin Server, and I saw a lot of logs, which was unuseful for me, like:
2022-07-26 10:28:32,212 INFO 1 -- [Thread-17317] _internal: 172.18.0.1 - - [26/Jul/2022 10:28:32] "GET /actuator/loggers HTTP/1.1" 200 -
I turned off the loggers on the site, and I want to setup in PyCharm if the server do any processes with any endpoints, then send back messages, like: 'System started xy process', 'System stopped xy process.' etc.
Could you please help how can I set it up in PyCharm?
Thanks!
I want to have a custom error page for internal errors. Right know i'm testing it by restarting my database service and trying to load a page before it connects again. The queries return an OperationalError
I've disabled debug mode so that i don't get the stack trace but instead of the custom 500 page I get the default text on white background.
I'm not sure if this is the correct way to link a sqlalchemy error to the custom 500 page. If I don't define a event listener for sqlalchemy the flask error handler won't get called.
#db.event.listens_for(db.engine, "handle_error")
def handle_exception(context):
internal_error(context.original_exception)
#app.errorhandler(500)
def internal_error(error):
db.session.rollback()
return render_template('500.html'), 500
Removing the sqlalchemy event or reraising the error there also doesn't work.
127.0.0.1 - - [11/Jul/2019 17:29:04] "GET /products HTTP/1.1" 500 -
Error on request:
...stack strace...
sqlalchemy.exc.OperationalError: (psycopg2.errors.AdminShutdown) terminating connection due to administrator command
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
UPDATE: Now that I thik about it. I don't think is has anything to do with sqlalchemy. I tried dividing by zero to see if I will get my custom page but the same thing happens.
How can I view log messages on Google Cloud?: https://console.cloud.google.com/logs
This is what I see in the terminal when I run dev_appserver.py (locally running):
INFO 2016-05-16 14:00:45,118 module.py:787] default: "GET /static/images/contact.png HTTP/1.1" 304 -
INFO 2016-05-16 14:00:45,128 module.py:787] default: "GET /static/images/email.png HTTP/1.1" 304 -
INFO 2016-05-16 14:00:45,136 module.py:787] default: "GET /static/images/phone.png HTTP/1.1" 304 -
INFO 2016-05-16 14:00:45,487 basehandler.py:19] entering basehandler.py
INFO 2016-05-16 14:00:45,516 module.py:787] default: "GET /static/images/logo-349x209.png HTTP/1.1" 304 -
INFO 2016-05-16 14:00:45,562 requesthandlers.py:26] entering requesthandlers.py
INFO 2016-05-16 14:00:45,563 app.py:28] entering app.py
INFO 2016-05-16 14:00:45,563 app.py:198] Using development database
Both application log messages and request logging is displayed.
However when I view the log of the same code deployed I can only see the requests being logged:
The code I'm using to generate application log messages is something like:
import logging
logger = logging.getLogger("someLogger")
logger.info("entering app.py")
But I've also tried using logging.info(...) directly with the same results.
I've tried finding an answer to this in various resources but I've come up empty-handed, most refer to how to set log level when developing locally.
I'm guessing that I need to enable some setting in order to view application logs on Google Cloud Logs.
Resources that I've looked at:
https://cloud.google.com/logging/docs/view/logs_viewer
https://cloud.google.com/appengine/docs/python/logs/
How to change the logging level of dev_appserver
How do I write to the console in Google App Engine?
Google App Engine - Can not find my logging messages
https://docs.python.org/3/howto/logging.html
App engine groups the logs by request. You need to expand the log using the triangle/pointer on the left of the request in the 'new' GAE log viewer.
Personally I prefer using the old GAE log viewer, but I am unsure how much longer it will be around:
https://appengine.google.com/logs?app_id=s~xxx
(This viewer shows request + logs and allows log expansion)
An easy way to integrate Google Cloud Platform logging into your Python code is to create a subclass from logging.StreamHandler. This way logging levels will also match those of Google Cloud Logging, enabling you to filter based on severity. This solution also works within Cloud Run containers.
Also you can just add this handler to any existing logger configuration, without needing to change current logging code.
import json
import logging
import os
import sys
from logging import StreamHandler
from flask import request
class GoogleCloudHandler(StreamHandler):
def __init__(self):
StreamHandler.__init__(self)
def emit(self, record):
msg = self.format(record)
# Get project_id from Cloud Run environment
project = os.environ.get('GOOGLE_CLOUD_PROJECT')
# Build structured log messages as an object.
global_log_fields = {}
trace_header = request.headers.get('X-Cloud-Trace-Context')
if trace_header and project:
trace = trace_header.split('/')
global_log_fields['logging.googleapis.com/trace'] = (
f"projects/{project}/traces/{trace[0]}")
# Complete a structured log entry.
entry = dict(severity=record.levelname, message=msg)
print(json.dumps(entry))
sys.stdout.flush()
A way to configure and use the handler could be:
def get_logger():
logger = logging.getLogger(__name__)
if not logger.handlers:
gcp_handler = GoogleCloudHandler()
gcp_handler.setLevel(logging.DEBUG)
gcp_formatter = logging.Formatter(
'%(levelname)s %(asctime)s [%(filename)s:%(funcName)s:%(lineno)d] %(message)s')
gcp_handler.setFormatter(gcp_formatter)
logger.addHandler(gcp_handler)
return logger
I'm just getting started with cloud endpoints by following the tutorial from
here.
Even when i copy paste the code exactly, I'm unable to look at the custom endpoin methods. When I run the project and type
http://localhost:8080/_ah/api/explorer
I get redirected to a blank api-explorer page. Here are the logs from the App Engine Launcher
INFO 2015-09-22 20:45:52,114 devappserver2.py:763] Skipping SDK update check.
INFO 2015-09-22 20:45:52,661 api_server.py:205] Starting API server at: http://localhost:53371
INFO 2015-09-22 20:45:52,676 dispatcher.py:197] Starting module "default" running at: http://localhost:8080`
INFO 2015-09-22 20:45:52,676 admin_server.py:118] Starting admin server at: http://localhost:8000
INFO 2015-09-22 20:46:08,792 module.py:808] default: "GET /_ah/api/explorer HTTP/1.1" 302 -
INFO 2015-09-22 20:46:12,805 module.py:808] default: "GET /_ah/api/explorer HTTP/1.1" 302 -
INFO 2015-09-22 20:46:17,711 module.py:808] default: "GET /_ah/api/static/proxy.html?jsh=m%3B%2F_%2Fscs%2Fapps-static%2F_%2Fjs%2Fk%3Doz.gapi.en.z0-pPDgHPQw.O%2Fm%3D__features__%2Fam%3DAQ%2Frt%3Dj%2Fd%3D1%2Ft%3Dzcms%2Frs%3DAGLTcCO4-ZDXf1JrE4AXDItAeFdjXKLW8w HTTP/1.1" 200 7690
The app.yaml and python files are exactly as described in the tutorial.
Can someone please tell me what exactly is going on and how to fix it.
Is there any smart way to find bottlenecks in business logic. For example, we have application, that have one view that doing HttpResponse('1') in big project. We are sure, that no SQL queries in middlewares exists. But HttpResponse working really slow(50 rps vs 200 rps on clear django project).
What reasons can be?
How to find bottlenecks in this case?
Also we know, that in clear project less than 1 Mb of memory used for objects on each request, and in our project - more than 2Mb. How to find these objects?
The debug toolbar works well, but I also like running django-devserver. It can give you more information than you can process sometimes.
DEVSERVER_MODULES = (
'devserver.modules.sql.SQLRealTimeModule',
'devserver.modules.sql.SQLSummaryModule',
'devserver.modules.profile.ProfileSummaryModule',
# Modules not enabled by default
'devserver.modules.ajax.AjaxDumpModule',
#'devserver.modules.profile.MemoryUseModule',
'devserver.modules.cache.CacheSummaryModule',
#'devserver.modules.profile.LineProfilerModule',
)
This is what modules I have turned on, and one hit to the admin page after start:
Django version 1.3.1, using settings 'myproject.settings' Running django-devserver 0.3.1 Threaded django server is running at http://127.0.0.1:8000/ Quit the server with CTRL-BREAK.
[sql] SELECT ...
FROM "auth_message"
WHERE "auth_message"."user_id" = 1
[sql] SELECT ...
FROM "django_admin_log"
INNER JOIN "auth_user" ON ("django_admin_log"."user_id" = "auth_user"."id")
LEFT OUTER JOIN "django_content_type" ON ("django_admin_log"."content_typ_id" = "django_content_type"."id")
WHERE "django_admin_log"."user_id" = 1
ORDER BY "django_admin_log"."action_time" DESC LIMIT 10
[sql] 4 queries with 0 duplicates
[profile] Total time to render was 0.54s
[cache] 0 calls made with a 100% hit percentage (0 misses) [30/Nov/2011 08:36:34] "GET /admin/ HTTP/1.1" 200 21667 (time: 0.69s; sql: 0ms (4q))
[sql] SELECT ...
FROM "django_flatpage"
INNER JOIN "django_flatpage_sites" ON ("django_flatpage"."id" = "django_fatpage_sites"."flatpage_id")
WHERE ("django_flatpage"."url" = /favicon.ico/
AND "django_flatpage_sites"."site_id" = 1)
[sql] 1 queries with 0 duplicates
[profile] Total time to render was 0.02s
[cache] 0 calls made with a 100% hit percentage (0 misses) [30/Nov/2011 08:36:34] "GET /favicon.ico/ HTTP/1.1" 404 2587 (time:
0.89s; sql: 0ms (1q))
Do you use the django debug toolbar ? You could find what queries are run with it, middleware or not.
How do you monitor the performance of the view ?
Are there much more users in the big project than in the fresh one ?
I guess your bottle neck is not in your or the django code. What webserver do you use, and how many requests are handled by the worker processes?
If you use mod_wsgi be sure to have enough worker processes and that maximum-requests is high.
And of course be sure that settings.DEBUG is not set.
Apache logs can include the request process time in microseconds: http://httpd.apache.org/docs/current/mod/mod_log_config.html check for %D
Check in your middleware how long the interpreter is inside your+django code.
# Middleware to check how long the request was in the wsgi queue:
class FooMiddleware:
def process_request(self, request):
...
queue_start=request.META.get('HTTP_X_QUEUE_START', None)
if queue_start is not None:
# How long was the request waiting in the wsgi queue?
# In Apache Config:
# RequestHeader add X-Queue-Start "%t" (in <VirtualHost>)
queue_start = int(queue_start[2:])/1000000.0
wait_in_queue=time.time()-queue_start
if wait_in_queue>1:
logging.error('Request was too long (%.3fs) in wsgi-queue: %s' % (
wait_in_queue, request.build_absolute_uri()))
You could try New Relic and see if it helpful in narrowing down problem area.
http://newrelic.com/python
http://blog.newrelic.com/2011/11/08/new-relic-supports-python/
Good thing is that you can use it on a production application where Django debug toolbar you cant.