App Engine giving errors about outdated files - python

Just to be extremely clear, here's some initial information about my setup:
Platform: Google App Engine
Environment Type: Standard
Runtime: Python 3
Python Framework: Flask
Web Server: Gunicorn, 2 workers
It's a flask app running on GAE standard python3 runtime.
Relevant File Structure:
- api.py
- services/
---- HighLow.py
As you can hopefully see from the file structure, the main flask app file is api.py.
api.py then imports services/HighLow.py
Relevant Code From HighLow.py:
def get_today_for_user(self, uid):
#Connect to MySQL
conn = pymysql.connect(self.host, self.username, self.password, self.database, cursorclass=pymysql.cursors.DictCursor)
cursor = conn.cursor()
uid = bleach.clean(uid)
cursor.execute("SELECT * FROM highlows WHERE uid='{}' AND DATE(_timestamp) = CURDATE();")
highlow = cursor.fetchone()
conn.commit()
conn.close()
if highlow == None:
return {
"high":"",
"low":"",
"total_likes": 0,
"high_image": "",
"low_image": ""
}
return highlow
If you want to look at the full code for better context, you can view it on our GitHub repo: https://github.com/highlowapp/highlowbackend
The Problem:
This setup has been working fine for a while now, and I was perfectly happy with it. However, I recently started getting this error from GCP's Error Reporting:
File "/srv/services/HighLow.py", line 380: 'low_image': ""
at <module> (/srv/api.py:8)
at <module> (/srv/wsgi.py:1)
at import_app (/env/lib/python3.7/site-packages/gunicorn/util.py:350)
at load_wsgiapp (/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py:41)
at load (/env/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py:52)
at wsgi (/env/lib/python3.7/site-packages/gunicorn/app/base.py:67)
at load_wsgi (/env/lib/python3.7/site-packages/gunicorn/workers/base.py:138)
at init_process (/env/lib/python3.7/site-packages/gunicorn/workers/base.py:129)
at spawn_worker (/env/lib/python3.7/site-packages/gunicorn/arbiter.py:583)
This error confuses me for a couple of reasons:
It doesn't give any kind of exception, it just "announces" that line 380 says 'low_image': "".
When I update the code to change that line, it gives me the same error. In other words, it says that line 380 is: 'low_image': "", when in reality (you can verify this on the GitHub repository), it's: "low_image": "" (with double quotes). I've also tried simply deleting the line, but the error continues to appear.
I've tried deleting app engine's storage bucket and re-running gcloud app deploy, but nothing has changed that error message at all.
My Question
Why am I getting this error message, and how do I fix it?
EDIT:
As I said above, I already tried deleting the staging bucket, and that didn't work.
Also, I used the "source" tool with Stackdriver to verify that the code has been updated.
Finally, I tried creating a new App Engine app in a separate project, and the code worked...except that I had an identical error in my code (maybe I'm just forgetting commas now?), which made the same error occur in a different file.
So I updated the file, redeployed, and voila! my app works!
However, I need it to work in the current project, not the test project. I need a way to clear everything that GAE knows about anything, and start all over. Is there a way to do this? Or does someone know why I'm having problems in the first place?

Ok, I finally figured out a workaround, and thought I'd share for anyone else who encounters this issue.
You know how tech support people always tell you to try turning it off and then back on? Let me tell you, it works.
I went to App Engine -> Settings -> Disable Application, and let it disable, and then re-enabled it with App Engine -> Settings -> Enable Application.
I guess it just needed to "reboot".

Related

Python not getting called(?) in React/Flask Project

Summary: I am trying to teach myself Flask and React with the tools I have at hand during the pandemic. I am hampered by the lack of a good React editor for remote files, I suspect I may not be seeing error messages such an editor would be showing me, and I suspect that the python script I am trying to invoke is never getting touched.
Any help, including how to better see error messages, or what might be mis-configured, are desperately appreciated.
What I have:
A command line only interface to an Ubuntu 18.04 container. This means: no remote desktop, no GUI, no specialized react editor. I cannot do anything about this.
React server up and running on container port 3000, accessible to the outside world with a path prefix per this question and answer. Meaning, I can run react from the text terminal command line, edit react project files in another window with vi, point a browser to it from well outside the system and it works.
Flask server up and running on container port 3001 (port 5000 already doing something else) and React server configured with port 3001 as a proxy. God help me, I am using the DANGEROUSLY_DISABLE_HOST_CHECK option to get that to work. No special consideration for port prefixing or port forwarding-- this seems all internal to the container, so I don't think any special handling is necessary.
A tutorial project as per this site. I will include the two relevant code files below.
The results of this are:
No errors in either of the terminals where I invoked the react and flask servers.
The web page displays, but improperly: Rather than displaying the correct time, the page displays only whatever I pre-loaded into currentTime.
I am highly skeptical that the python script is ever running at all: If I mangle the api.py file with egregious syntactical errors... no change in behavior. If I mangle the App.js with errors, the system dies. If I mangle the App.js with the specific error of fetch('/garbage').then... rather than fetch('/time').then... there are still no errors, which I find HIGHLY suspicious.
This makes me think I may not be seeing some critical error or warning messages.
My suspicion was that this might be some type of CORS issue, but two attempts at adapting solutions to this question failed.
Code from App.js:
import React, { useState, useEffect } from 'react';
import logo from './logo.svg';
import './App.css';
function App() {
const [currentTime, setCurrentTime] = useState(2);
useEffect(() => {
fetch('/time').then(res => res.json()).then(data => {
setCurrentTime(data.time);
});
}, []);
return (
<div className="App">
<header className="App-header">
<img src={logo} className="App-logo" alt="logo" />
<p>
Edit <code>src/App.js</code> and save to reload.
</p>
<a
className="App-link"
href="https://reactjs.org"
target="_blank"
rel="noopener noreferrer"
>
Learn React
</a>
<p>The current time is {currentTime}.</p>
</header>
</div>
);
}
export default App;
Code from api.py (not including CORS attempts):
import time
from flask import Flask
app = Flask(__name__)
#app.route('/time')
def get_current_time():
return {'time': time.time()}
UPDATE:
On the advice of this answer, I added some console logging statements and looked at the console logging in Chrome.
App.js:11 GET http://<CORRECT_ADDRESS>/time 404 (Not Found)
(anonymous) # App.js:11
commitHookEffectListMount # react-dom.development.js:19731
commitPassiveHookEffects # react-dom.development.js:19769
callCallback # react-dom.development.js:188
invokeGuardedCallbackDev # react-dom.development.js:237
invokeGuardedCallback # react-dom.development.js:292
flushPassiveEffectsImpl # react-dom.development.js:22853
unstable_runWithPriority # scheduler.development.js:653
runWithPriority$1 # react-dom.development.js:11039
flushPassiveEffects # react-dom.development.js:22820
(anonymous) # react-dom.development.js:22699
workLoop # scheduler.development.js:597
flushWork # scheduler.development.js:552
performWorkUntilDeadline # scheduler.development.js:164
Note 1: Port 3001 is not included, which seems wrong.
Note 2: The path prefix is not included, which seems correct.
I would have expected this to be:
App.js:11 GET http://<CORRECT_ADDRESS>:3001/time
If the port, and not the path prefix, needs to be included in that call, how do I arrange that?
Flask is running on port 3001, and the last line of package.json is:
"proxy": "http://localhost:3001"
Further Update:
using fetch('http://<CORRECT_ADDRESS>:3001/time') now yields the following console errors:
WebSocket connection to 'ws://<CORRECT_ADDRESS>/sockjs-node' failed: Error during WebSocket handshake: Unexpected response code: 404
The development server has disconnected.
Refresh the page if necessary.
GET http://<CORRECT_ADDRESS>:3001/time net::ERR_CONNECTION_TIMED_OUT
Uncaught (in promise) TypeError: Failed to fetch
So the GET seems to be well-formed, now, but times out due to the failed WebSockets call, which may or may not be correctly formed.
You may be seeing an infinite loop, where state is not having a chance to update.
Try this:
#app.route('/time')
def get_current_time():
print( time.time() )
return {'time': time.time()}
Is /time getting hits? Because your useEffect() code will run every time the state changes, which is every time it runs: infinite loop.
If /time is not getting hits, then your problem is in that relative path. Try this:
useEffect(() => {
console.log('fetching');
fetch('/time').then(res => res.json()).then(data => {
console.log('fetched:' + data);
setCurrentTime(data.time);
});
}, []);
Next, check your ports. It sounds like you may have React/Webpack running on 3001, and the server running on 5000.
Try these:
http://localhost:5000/time
http://localhost:3000/time
http://localhost:3001/time

Angular router not engaging with Flask server using ng build

When using Angular 5 with Flask and ng build --aot --watch, I am having major issues getting Angular's routing and refreshing the page to work.
I have come across many attempts to fix this, but none of them when applied to my project work (and the example projects provided when I try it there also do not seem to work).
What happens (depending on the thing I try) are these:
It can't find the path at all and I get a Jinja2 error
It can't find the document for localhost:5015/ (the port I'm using) and I get a 404 and a white screen
It loads index.html, but upon refresh it gets a 404 (implying that the browser is attempting to fetch the page from the server, not using the Angular routing system)
I get the same issue as this one: Syntax Error in Angular App: Unexpected token <
I am under the impression this is a normal issue with all SPAs and that the solution is to always redirect the server to index.html. We have done this (AFAIK), and attempted multiple ways to do this:
Our structure is roughly the following:
- ProjectDir
- reverseProxy_Calls
- scripts
- swagger_yaml_definitions
- templates
- e2e
- dist
- src
- app
And our server.py contains this (among other stuff)
Swagger(app)
app.template_folder = "./templates/dist/"
app.static_folder = "./templates/dist/"
app.static_path = "./templates/dist/"
app.static_url_path = "./templates/dist/"
app.instance_path = "./templates/dist/"
config = settings.config()
config.read("settings.cfg")
#app.route('/', defaults={'path': ''}, methods=['GET', 'POST'])
#app.route('/<path:path>', methods=['GET', 'POST'])
def get_resource(path): # pragma: no cover
# If we are devmode, proxy through flask.
if config.getKey("devmode") == True:
url = "http://localhost:%s/%s" % (config.getKey('ng_port'), path)
j = requests.get(url, stream=True, params=request.args)
return make_response(j.text)
# return the results
# return render_template('dist/index.html')
# return app.send_static_file("index.html")
return send_from_directory('dist/', path)
if __name__ == '__main__':
app.run(debug=True, port=int(config.getKey("port"))) # , host='0.0.0.0')
^ A variety of ways here show various attempts to handle this.
I have also looked at and tried projects that use variations of the following:
Flask Cli
Flask-Script
Flask-Bootstrap
None of them have solved this problem when I have tried them.
I have been told that using the hashRoute location strategy in Angular rather than the History PushState strategy will also work, however when I attempted to do that it had the same behavior of losing the page upon refresh (additionally we wish to use server side rendering in the future, and hashRoute location strategy prevents this possibility, among many other downsides).
Possibly related is that Flask seems to reload or re-render the page when it does find the index.html and routes initially, wiping anything printed to the browser dev console. I have seen this in some of the flask examples, but not all, and I'm uncertain what causes this.
When running via ng serve --aot --prod, things work exactly correctly and as expected. It's only when serving via Flask that things break down.
Apologies for the long question and thank you for your time in advance.
I am under the impression this is a normal issue with all SPAs and
that the solution is to always redirect the server to index.html. We
have done this (AFAIK), and attempted multiple ways to do this
You also need to redirect all HTTP errors to index.html.

flask how to log all warning exceptions (full stack trace) to file

So I am trying to build a restful API using flask, served up by apache on centos (httpd).
Basic API calls work just fine but I am not making much progress on the more advanced aspects because every time it fails I just get an HTTP 500 response which is completely useless for troubleshooting and I have no server-side logs to look at. I am literally trying to solve this through trial and error and it is making me bang my head against the wall.
In order to make any progress on this project I need to setup some basic error logging, but I do not understand the documentation or existing threads about this. It is completely over my head.
What I want to do is have flask write out all warnings and exceptions generated by my application to a specific file (it can be in the app directory to keep it simple).
I am looking for the simplest, easiest, least mind bendy way of doing this... suggestions?
Here is a very simplified version of my app... it shows the basic structure I am using, so please use that for reference.
from flask import Flask, jsonify, request
from flask_restful import reqparse, abort, Resource, Api
app = Flask(__name__)
api = Api(app)
class fetchTicket(Resource):
def post(self):
request_data = request.get_json(force=True)
r_ticket_id = request_data['ticket_id']
return jsonify(ticket_id=r_ticket_id)
api.add_resource(fetchTicket, '/ticket/fetch')
if __name__ == "__main__":
import logging
from logging.handlers import FileHandler
app.debug = True
file_handler = FileHandler("/var/www/project_folder/error.log")
file_handler.setLevel(logging.DEBUG)
app.logger.addHandler(file_handler)
app.run()
But when I run the above code no error.log file is created. I am not sure what I am doing wrong.
Note: I did set the folder permissions so that the apache user has access to write to the directory where the log file should go, as per http://fideloper.com/user-group-permissions-chmod-apache but it did not help so I don't think it is a permissions issue.
You’ll need to explicitly include the stack trace when logging, using the extra kwarg.
logger.exception('Probably something went wrong', extra={'stack': True})

Detect Errors to Improve Error Logging in Google App Engine Python

Is there an elegant way to detect whether my google app engine application has had an error during a request to the application, and write out a much more detailed log when it does?
I have considerable "debug" level logging, but I would like to remove all that and only display the debug information when an error occurs.
If you are using webapp2 I recommend do it during a request with the next code:
app = webapp2.WSGIApplication(debug = os.environ['SERVER_SOFTWARE'].startswith('Dev'), config=config.webapp2_config)
app.error_handlers[404] = handle_error
app.error_handlers[500] = handle_error
routes.add_routes(app)
You can see an example for that handle_error Here :
https://github.com/coto/gae-boilerplate/blob/master/boilerplate/lib/basehandler.py#L70
You need to start from the LogService API docs; Fetch and filter some request_logs (remember to include app_logs); For any req_log you can get the related app_logs. Any app_log have three property (time, level and message). What you looking for is message property.
req_logs = logservice.fetch(start_time=time.time() - 3600,
end_time=time.time(),
minimum_log_level=logservice.LOG_LEVEL_ERROR,
include_app_logs=True)
for req_log in req_logs:
for app_log in req_log.app_logs:
print app_log.message

Django + FastCGI - randomly raising OperationalError

I'm running a Django application. Had it under Apache + mod_python before, and it was all OK. Switched to Lighttpd + FastCGI. Now I randomly get the following exception (neither the place nor the time where it appears seem to be predictable). Since it's random, and it appears only after switching to FastCGI, I assume it has something to do with some settings.
Found a few results when googleing, but they seem to be related to setting maxrequests=1. However, I use the default, which is 0.
Any ideas where to look for?
PS. I'm using PostgreSQL. Might be related to that as well, since the exception appears when making a database query.
File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 86, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python2.6/site-packages/django/contrib/admin/sites.py", line 140, in root
if not self.has_permission(request):
File "/usr/lib/python2.6/site-packages/django/contrib/admin/sites.py", line 99, in has_permission
return request.user.is_authenticated() and request.user.is_staff
File "/usr/lib/python2.6/site-packages/django/contrib/auth/middleware.py", line 5, in __get__
request._cached_user = get_user(request)
File "/usr/lib/python2.6/site-packages/django/contrib/auth/__init__.py", line 83, in get_user
user_id = request.session[SESSION_KEY]
File "/usr/lib/python2.6/site-packages/django/contrib/sessions/backends/base.py", line 46, in __getitem__
return self._session[key]
File "/usr/lib/python2.6/site-packages/django/contrib/sessions/backends/base.py", line 172, in _get_session
self._session_cache = self.load()
File "/usr/lib/python2.6/site-packages/django/contrib/sessions/backends/db.py", line 16, in load
expire_date__gt=datetime.datetime.now()
File "/usr/lib/python2.6/site-packages/django/db/models/manager.py", line 93, in get
return self.get_query_set().get(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 304, in get
num = len(clone)
File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 160, in __len__
self._result_cache = list(self.iterator())
File "/usr/lib/python2.6/site-packages/django/db/models/query.py", line 275, in iterator
for row in self.query.results_iter():
File "/usr/lib/python2.6/site-packages/django/db/models/sql/query.py", line 206, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/lib/python2.6/site-packages/django/db/models/sql/query.py", line 1734, in execute_sql
cursor.execute(sql, params)
OperationalError: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Possible solution: http://groups.google.com/group/django-users/browse_thread/thread/2c7421cdb9b99e48
Until recently I was curious to test
this on Django 1.1.1. Will this
exception be thrown again... surprise,
there it was again. It took me some
time to debug this, helpful hint was
that it only shows when (pre)forking.
So for those who getting randomly
those exceptions, I can say... fix
your code :) Ok.. seriously, there
are always few ways of doing this, so
let me firs explain where is a
problem first. If you access database
when any of your modules will import
as, e.g. reading configuration from
database then you will get this error.
When your fastcgi-prefork application
starts, first it imports all modules,
and only after this forks children.
If you have established db connection
during import all children processes
will have an exact copy of that
object. This connection is being
closed at the end of request phase
(request_finished signal). So first
child which will be called to process
your request, will close this
connection. But what will happen to
the rest of the child processes? They
will believe that they have open and
presumably working connection to the
db, so any db operation will cause an
exception. Why this is not showing in
threaded execution model? I suppose
because threads are using same object
and know when any other thread is
closing connection. How to fix this?
Best way is to fix your code... but
this can be difficult sometimes.
Other option, in my opinion quite
clean, is to write somewhere in your
application small piece of code:
from django.db import connection
from django.core import signals
def close_connection(**kwargs):
connection.close()
signals.request_started.connect(close_connection)
Not ideal thought, connecting twice to the DB is a workaround at best.
Possible solution: using connection pooling (pgpool, pgbouncer), so you have DB connections pooled and stable, and handed fast to your FCGI daemons.
The problem is that this triggers another bug, psycopg2 raising an InterfaceError because it's trying to disconnect twice (pgbouncer already handled this).
Now the culprit is Django signal request_finished triggering connection.close(), and failing loud even if it was already disconnected. I don't think this behavior is desired, as if the request already finished, we don't care about the DB connection anymore. A patch for correcting this should be simple.
The relevant traceback:
/usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/core/handlers/wsgi.py in __call__(self=<django.core.handlers.wsgi.WSGIHandler object at 0x24fb210>, environ={'AUTH_TYPE': 'Basic', 'DOCUMENT_ROOT': '/storage/test', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTPS': 'off', 'HTTP_ACCEPT': 'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate', 'HTTP_AUTHORIZATION': 'Basic dGVzdGU6c3VjZXNzbw==', 'HTTP_CONNECTION': 'keep-alive', 'HTTP_COOKIE': '__utma=175602209.1371964931.1269354495.126938948...none); sessionid=a1990f0d8d32c78a285489586c510e8c', 'HTTP_HOST': 'www.rede-colibri.com', ...}, start_response=<function start_response at 0x24f87d0>)
246 response = self.apply_response_fixes(request, response)
247 finally:
248 signals.request_finished.send(sender=self.__class__)
249
250 try:
global signals = <module 'django.core.signals' from '/usr/local/l.../Django-1.1.1-py2.6.egg/django/core/signals.pyc'>, signals.request_finished = <django.dispatch.dispatcher.Signal object at 0x1975710>, signals.request_finished.send = <bound method Signal.send of <django.dispatch.dispatcher.Signal object at 0x1975710>>, sender undefined, self = <django.core.handlers.wsgi.WSGIHandler object at 0x24fb210>, self.__class__ = <class 'django.core.handlers.wsgi.WSGIHandler'>
/usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/dispatch/dispatcher.py in send(self=<django.dispatch.dispatcher.Signal object at 0x1975710>, sender=<class 'django.core.handlers.wsgi.WSGIHandler'>, **named={})
164
165 for receiver in self._live_receivers(_make_id(sender)):
166 response = receiver(signal=self, sender=sender, **named)
167 responses.append((receiver, response))
168 return responses
response undefined, receiver = <function close_connection at 0x197b050>, signal undefined, self = <django.dispatch.dispatcher.Signal object at 0x1975710>, sender = <class 'django.core.handlers.wsgi.WSGIHandler'>, named = {}
/usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/db/__init__.py in close_connection(**kwargs={'sender': <class 'django.core.handlers.wsgi.WSGIHandler'>, 'signal': <django.dispatch.dispatcher.Signal object at 0x1975710>})
63 # when a Django request is finished.
64 def close_connection(**kwargs):
65 connection.close()
66 signals.request_finished.connect(close_connection)
67
global connection = <django.db.backends.postgresql_psycopg2.base.DatabaseWrapper object at 0x17b14c8>, connection.close = <bound method DatabaseWrapper.close of <django.d...ycopg2.base.DatabaseWrapper object at 0x17b14c8>>
/usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/db/backends/__init__.py in close(self=<django.db.backends.postgresql_psycopg2.base.DatabaseWrapper object at 0x17b14c8>)
74 def close(self):
75 if self.connection is not None:
76 self.connection.close()
77 self.connection = None
78
self = <django.db.backends.postgresql_psycopg2.base.DatabaseWrapper object at 0x17b14c8>, self.connection = <connection object at 0x1f80870; dsn: 'dbname=co...st=127.0.0.1 port=6432 user=postgres', closed: 2>, self.connection.close = <built-in method close of psycopg2._psycopg.connection object at 0x1f80870>
Exception handling here could add more leniency:
/usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/db/__init__.py
63 # when a Django request is finished.
64 def close_connection(**kwargs):
65 connection.close()
66 signals.request_finished.connect(close_connection)
Or it could be handled better on psycopg2, so to not throw fatal errors if all we're trying to do is disconnect and it already is:
/usr/local/lib/python2.6/dist-packages/Django-1.1.1-py2.6.egg/django/db/backends/__init__.py
74 def close(self):
75 if self.connection is not None:
76 self.connection.close()
77 self.connection = None
Other than that, I'm short on ideas.
In the switch, did you change PostgreSQL client/server versions?
I have seen similar problems with php+mysql, and the culprit was an incompatibility between the client/server versions (even though they had the same major version!)
Smells like a possible threading problem. Django is not guaranteed thread-safe although the in-file docs seem to indicate that Django/FCGI can be run that way. Try running with prefork and then beat the crap out of the server. If the problem goes away ...
Maybe the PYTHONPATH and PATH environment variable is different for both setups (Apache+mod_python and lighttpd + FastCGI).
In the end I switched back to Apache + mod_python (I was having other random errors with fcgi, besides this one) and everything is good and stable now.
The question still remains open. In case anybody has this problem in the future and solves it they can record the solution here for future reference. :)
I fixed a similar issue when using a geodjango model that was not using the default ORM for one of its functions. When I added a line to manually close the connection the error went away.
http://code.djangoproject.com/ticket/9437
I still see the error randomly (~50% of requests) when doing stuff with user login/sessions however.
I went through the same problem recently (lighttpd, fastcgi & postgre). Searched for a solution for days without success, and as a last resort switched to mysql. The problem is gone.
Why not storing session in cache?
Set
SESSION_ENGINE = "django.contrib.sessions.backends.cache"
Also you can try use postgres with pgbouncer (postgres - prefork server and don't like many connects/disconnects per time), but firstly check your postgresql.log.
Another version - you have many records in session tables and django-admin.py cleanup can help.
The problem could be mainly with Imports. Atleast thats what happened to me.
I wrote my own solution after finding nothing from the web. Please check my blogpost here: Simple Python Utility to check all Imports in your project
Ofcourse this will only help you to get to the solution of the original issue pretty quickly and not the actual solution for your problem by itself.
Change from method=prefork to method=threaded solved the problem for me.
I try to give an answer to this even if I'am not using django but pyramid as the framework. I was running into this problem since a long time. Problem was, that it was really difficult to produce this error for tests... Anyway. Finally I solved it by digging through the whole stuff of sessions, scoped sessions, instances of sessions, engines and connections etc. I found this:
http://docs.sqlalchemy.org/en/rel_0_7/core/pooling.html#disconnect-handling-pessimistic
This approach simply adds a listener to the connection pool of the engine. In the listener a static select is queried to the database. If it fails the pool try to establish a new connection to the database before it fails at all. Important: This happens before any other stuff is thrown to the database. So it is possible to pre check connection what prevents the rest of your code from failing.
This is not a clean solution since it don't solve the error itself but it works like a charm. Hope this helps someone.
An applicable quote:
"2019 anyone?"
- half of YouTube comments, circa 2019
If anyone is still dealing with this, make sure your app is "eagerly forking" such that your Python DB driver (psycopg2 for me) isn't sharing resources between processes.
I solved this issue on uWSGI by adding the lazy-apps = true option, which causes is to fork app processes right out of the gate, rather than waiting for copy-on-write. I imagine other WSGI / FastCGI hosts have similar options.
Have you considered downgrading to Python 2.5.x (2.5.4 specifically)? I don't think Django would be considered mature on Python 2.6 since there are some backwards incompatible changes. However, I doubt this will fix your problem.
Also, Django 1.0.2 fixed some nefarious little bugs so make sure you're running that. This very well could fix your problem.

Categories