Edit:
After doing some further investigation, the delay seems to be more Django than the Cloud SQL Proxy.
I added a couple of print statements at the start and end of a view, and they print instantly when the request is made, but it takes a further 60 seconds for the page to load.
I've stripped back the template files to include only the bare bones, removing most scripts and static resources and it's still pretty similar.
Changing my view to return a simple HttpResponse('Done') cuts the time drastically.
Whilst developing locally I am using Django to serve the static files as described in the docs. Again, I don't have this issue with other projects though.
Original Post:
I've recently noticed my Django application is incredibly slow to connect to my Google Cloud SQL database when using the Cloud SQL Proxy in my local development environment.
The initial connection takes 2-3 minutes, then 60 seconds per request thereafter. This applies when performing migrations or running the development server. Eventually the request completes.
I've tried scaling up the database but to no effect (it's relatively small anyway). Database version is MySQL 5.7 with machine type db-n1-standard-1. Previously I've used Django Channels but have since removed all references to this.
The Middleware and settings.py are relatively standard and identical to another Django app that connects in an instant.
The live site also connects very fast without any issues.
Python version is 3.6 w/ Django 2.1.4 and mysqlclient 1.3.14.
My database settings are defined as:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'NAME': os.getenv('DB_NAME'),
'USER': os.getenv('DB_USER'),
'PASSWORD': os.getenv('DB_PASSWORD'),
'PORT': '3306',
}
}
DATABASES['default']['HOST'] = os.getenv('DB_HOST')
if os.getenv('GAE_INSTANCE'):
pass
else:
DATABASES['default']['HOST'] = '127.0.0.1'
Using environment variables or not doesn't seem to make a difference.
I'm starting the Cloud SQL Proxy via ./cloud_sql_proxy -instances="my-project:europe-west1:my-project-instance"=tcp:3306.
After invoking the proxy via the command line I see Ready for new connections. Running python manage.py runserver shows New connection for "my-project:europe-west1:my-project-instance" but then takes an age before I see Starting development server at http://127.0.0.1:8000/.
I'm also noticing several errors in Stackdriver:
_mysql_exceptions.OperationalError: (2006, "Lost connection to MySQL server at 'reading initial communication packet', system error: 95")
django.db.utils.OperationalError: (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 95")
AttributeError: 'SessionStore' object has no attribute '_session_cache'
These appear - or don't - from time to time without changing any settings.
I've read they may be an access rights issue but the connection is eventually made, it's just incredibly slow. I'm authorising via the Google Cloud SDK, which seems to work fine.
Eventually I found that the main source of the delay was a recursive function being called in one of my admin forms (which delayed the initial startup) and context processors (which delayed each load). After removing it, the pages loaded without issue. It worked fine when deployed to App Engine or when using a test/local SQLite database, though, which is what made debugging a little harder.
Related
I have a Flask App running on an Ubuntu WebApp on Azure. Every morning my queries to the app fail with the below error:
sqlalchemy.exc.OperationalError: (pyodbc.OperationalError) ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: Error code 0x68 (104) (SQLExecDirectW)')
I am using SQLAlchemy ORM to query my Azure SQL Server Instance. I believe that my connections are becoming stale for the following reasons.
It happens every morning after no one uses the app
After X many failed returns, it starts working, until the next morning.
However to make things more weird, when I check sys.dm_exec_sessions on the sql server, it does not show any active connections (outside of the one I'm executing to check).
In addition, when I run the dockerized app on my local and connect to the DB I get no such error.
If anyone has had a similar issue I'd love some insights, or at least a recommendation on where to drill down.
https://azure.github.io/AppService/2018/03/01/Deep-Dive-into-TCP-Connections-in-App-Service-Diagnostics.html
This link helped me, but the solution is only for Windows Apps, not Linux.
With help from #snakecharmerb:
The application was in-fact holding on to a pool of dead connections, setting pool_recycle to a greater time solved the issue.
engine = create_engine(
key, echo=False, future=True,
echo_pool=True,
# connection pool will log informational output such as when connections are invalidated.
pool_recycle=3600
# causes the pool to recycle connections after the given number of seconds has passed.
)
I am currently working with an online database at a local server (but I don't have physical access to the server). I want to use python (version 3.5) to connect to the database and work with the data.
Since the problem is pretty early in the developing process I have not tried too much yet. In my mind I would like it to work somehow like this:
import sqlite3 as sql
db = sql.connect('localhost:port', 'username', 'password', 'database.db')
I expect it to return the database connection object I could then work with.
You'll need public IP of the VM/server where the database is hosted.
db = sql.connect('<PUBLIC_IP:PORT>', '<USER_NAME>', '<PASSWORD>', '<DATABASE>')
will get your requirement done. If it does not work, and you get connection error,
make sure that the port is correct.
I have built a django website and a part of it is Microsoft authentication link.
When I upload the site to azure cloud and click on the "log in" link, I recieve the following error:
OperationalError at /login
database is locked
Request Method: GET
Request URL: http://bhkshield.azurewebsites.net/login
Django Version: 2.2.2
Exception Type: OperationalError
Exception Value:
database is locked
Exception Location: /home/site/wwwroot/antenv3.6/lib/python3.6/site-packages/django/db/backends/base/base.py in _commit, line 240
Python Executable: /usr/local/bin/python
Python Version: 3.6.7
Python Path:
['/usr/local/bin',
'/home/site/wwwroot',
'/home/site/wwwroot/antenv3.6/lib/python3.6/site-packages',
'/usr/local/lib/python36.zip',
'/usr/local/lib/python3.6',
'/usr/local/lib/python3.6/lib-dynload',
'/usr/local/lib/python3.6/site-packages']
Server time: Fri, 14 Jun 2019 13:19:22 +0000
I am using sqlite3 (setting.py code piece):
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
I don't understand why I get this error because I don't insert or commit anything to the database.
My website consists only of one page that has a sign in link (4 views: home, contex intialize, login and callback). That's it.
Just to mention, when I run the site locally, everything works. It stops working only after deployment.
Another weird thing is that I have uploaded something like this before to another site on azure and the login worked. For some reason, it doesn't work now and I have no idea why...
Has anyone encountered this type of error and can help?
If you need me to provide more files' content, let me know which files and I will.
It seems like a duplication of this question: OperationalError: database is locked.
From the documentation of Django:
https://docs.djangoproject.com/en/dev/ref/databases/#database-is-locked-errorsoption
SQLite is meant to be a lightweight database, and thus can’t support a
high level of concurrency. OperationalError: database is locked errors
indicate that your application is experiencing more concurrency than
sqlite can handle in default configuration. This error means that one
thread or process has an exclusive lock on the database connection and
another thread timed out waiting for the lock the be released.
Python’s SQLite wrapper has a default timeout value that determines
how long the second thread is allowed to wait on the lock before it
times out and raises the OperationalError: database is locked error.
If you’re getting this error, you can solve it by:
Switching to another database backend. At a certain point SQLite
becomes too “lite” for real-world applications, and these sorts of
concurrency errors indicate you’ve reached that point.
Rewriting your code to reduce concurrency and ensure that database
transactions are short-lived.
Increase the default timeout value by setting the timeout database
option
I have been developing a Django web app as well, and I chose a Azure SQL Server to be the database of the application. Everything has been working OK.
I'm completely new to Django. I have developed a simple web application in Django and hosted it on external server. And that web application uses default mysql database. Now, I want to switch to a secondary server if my primary server goes down.
Copying and running the same code is not the option.
Can anyone explain how do I do it along with an example ?
not a beginner topic or answer but, in a prototypical production-esque deployment with nginx/apache/django, you can use the nginx upstream module
it'll detect when a node is down and won't route requests to the down node, so you'll get the failover behavior you desire out of the box
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#upstream
I googled "nginx upstream module failover" and the second result had some examples: https://serverfault.com/questions/140990/nginx-automatic-failover-load-balancing
I've built a small python REST service using Flask, with Flask-SQLAlchemy used for talking to the MySQL DB.
If I connect directly to the MySQL server everything is good, no problems at all. If I use HAproxy (handles HA/failover, though in this dev environment there is only one DB server) then I constantly get MySQL server has gone away errors if the application doesn't talk to the DB frequently enough.
My HAproxy client timeout is set to 50 seconds, so what I think is happening is it cuts the stream, but the application isn't aware and tries to make use of an invalid connection.
Is there a setting I should be using when using services like HAproxy?
Also it doesn't seem to reconnect automatically, but if I issue a request manually I get Can't reconnect until invalid transaction is rolled back, which is odd since it is just a select() call I'm making, so I don't think it is a commit() I'm missing - or should I be calling commit() after every ORM based query?
Just to tidy up this question with an answer I'll post what I (think I) did to solve the issues.
Problem 1: HAproxy
Either increase the HAproxy client timeout value (globally, or in the frontend definition) to a value longer than what MySQL is set to reset on (see this interesting and related SF question)
Or set SQLALCHEMY_POOL_RECYCLE = 30 (30 in my case was less than HAproxy client timeout) in Flask's app.config so that when the DB is initialised it will pull in those settings and recycle connections before HAproxy cuts them itself. Similar to this issue on SO.
Problem 2: Can't reconnect until invalid transaction is rolled back
I believe I fixed this by tweaking the way the DB is initialised and imported across various modules. I basically now have a module that simply has:
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()
Then in my main application factory I simply:
from common.database import db
db.init_app(app)
Also since I wanted to easily load table structures automatically I initialised the metadata binds within the app context, and I think it was this which cleanly handled the commit() issue/error I was getting, as I believe the database sessions are now being correctly terminated after each request.
with app.app_context():
# Setup DB binding
db.metadata.bind = db.engine