I'm having concurrency problems leading to PK violations in flask-sqlalchemy with MySQL using uwsgi with two worker threads and nginx.
I'm using keys that come from the payload. If the contact comes in with a key that exists in the database, update the record, if not, create a new record.
Here is the setup and what I think is happening.
#project/__init__.py
app = Flask(__name__)
db = SQLAlchemy(app)
from project import views, models
#project/views.py
from project import db
#app.route('/receive', methods = ['POST'])
def receive():
#Check to see if the contact exists in the database
contact = models.Contact.getIssue(contact_payload['id'])
if contact is None:
#If not in the DB, create a new contact
new_contact = models.Contact(contact_payload)
db.session.merge(new_contact)
else:
#If in the DB, create an updated representation of this contact
updated_issue = contact.updateContact(contact_payload)
db.session.merge(updated_issue)
...Some other stuff...
#Commit to DB
db.session.commit()
#Respond
resp = jsonify({'Result' : 'Success'})
resp.status_code = 200
return resp
The issue comes when we receive two requests for the same contact at the exact same time (requestA is 12:10:49,063 and requestB is 12:10:49,066). One of the requests ends in a PK violation.
I suspect they're coming in on different worker threads and each request gets a scoped session (sessionA - requestA and sessionB - requestB) from flask-sqlalchemy.
I suspect both sessions contain nothing at the beginning of the requests that are now in the above code flow simultaneously.
requestA goes through the appropriate code flow and merges new_contact into sessionA.
At the same time requestB goes through the same code path where contact is None (because sessionA hasn't committed yet) and merges new_contact into sessionB.
Then sessionA commits before sessionB. When sessionB goes to commit, it gets the PK violation.
Can we do anything else other than catch the PK violation and react accordingly?
You have two options:
Catch the PK violation and react accordingly, like you already said.
Lock your transaction based on your id: this is more complicated, you need something to synchronize your locks, like redis. Take a look at python-redis-lock. It is just one option, the solution here is to avoid concurrency for a PK.
https://pypi.python.org/pypi/python-redis-lock
Related
I'm pretty new to asking questions on stack overflow and building a web app with flask so I apologize in advance if I say something that's incorrect.
I'm building a flask web app that handles scheduling appointments and naturally i'm using 3rd party libraries/wrappers to make API calls and to handle db queries (sqlite). As it is currently, my app doesn't use any kind of async code. I got thinking about scenarios where multiple users try to login or book an appointment, both involving db queries and api calls, and whether my application will block or not. Given that its all synchronous, if multiple users do try and login and book an appointment, will it block for some users while it handles other user requests, or is that all independent of my flask app and dependent on the WSGI server running the app? Any tips on how I can improve the performance of my app with regards to handling multiple requests?
Example route that multiple users try to visit at the same time:
...
#app.route('/signup/')
def signup():
# has a form that takes user input and submits it for processing in process_signup()
return render_template('signup.html')
#app.route('/process_signup/', methods=['POST'])
def process_signup():
form = request.forms
username = form['username']
password = form['password']
# insert values into customer db in a synchronous fashion
db = db() #gets the database
cursor = db.execute('insert into customers (username, password) values (?,?)', [username, password])
db.commit()
#send out a verification email using email api without any async
email_new_customer()
return redirect(url_for('signin'))
NOTE: I read that its better to have db queries and api calls handled asynchronously but not sure where to start or what kind of library compatible with flask will help me achieve that.
Thank you for your help and again I may have a big misunderstanding of how it all works as i'm very to new to this so I apologize.
After i clear my browser history the session key and data that was added to the table django_session (session engine is database-backed by mysql) for the session remains and for subsequent requests a new 'session key' is added. In a situation where users clear their history frequently this table will grow inadvertently. Using clearsessions command doesn't remove the rows because the expiry date has not yet been reached. How do i overcome this issue if the expiry date is large? Is there some setting wrongly set up?
Actually, there is a way but highly inefficient since it requires iterating over all of the existing Sessions. You can use signals to catch post_save of Session model and then delete all of the existing Sessions of this user except the new one (I am not sure if this is what you want, because it will delete all of the sessions, not just the cleared one) .
from django.contrib.sessions.models import Session
from django.db.models.signals import post_save
from django.dispatch import receiver
#receiver(post_save, sender=Session)
def save_session(sender, instance, **kwargs):
user_id = instance.get_decoded().get('_auth_user_id')
# Get all sessions excluding current session
sessions = Session.objects.exclude(session_key=instance.session_key)
# Iterate over all sessions and decode them
for session in sessions:
session_user_id = session.get_decoded().get('_auth_user_id')
# If the session belongs to user, delete it
if session_user_id == user_id:
session.delete()
My company has a Flask application that uses flask-login to handle user logins/logouts and sessions. We use the #login_required decorator to protect views. Our clients log via persistent users stored in a database. However, we want employees of our company to be able to log in to our clients' sites without popping up in their users lists. We verify our own authenticity by CAS login at our own CAS server and checking that the username belongs to our company.
We want the employees to simply be able to login without needing to be persisted in the database but flask-login requires a persisted user model.
Sorry for the troublesome use of words here, but I hope this is understadable.
Every authorization system should check user in some storage by some field and in usual cases return exist or has permisions.
So with flask-login you can implement it with next: user_loader/header_loader/request_loader and UserMixin with user_id.
Every request with login_required call *_loader to get UserMixin. So it can look like next:
#login_manager.request_loader
def loader(request):
identifier = get_identifier_from_request(request)
exist = check_on_cas_server(identifier)
if not exist:
return None
user = UserMixin()
user.id = get_specified_or_random_id(identifier, exist)
return user
Details you can found with https://flask-login.readthedocs.org/en/latest/.
I am using the Django Test client (django.test.Client) to run view tests. Upon attempting to use the Test Client on my index function that handles post requests for logins, it continually fails the test even though the authentication successfully occurs.
Heres my test:
def test_login(self):
response = self.client.post('/login/', {'username':'user', 'password':'pass'})
print response.content
self.assertIn(SESSION_KEY, self.client.session)
So the reason i know the login process successfully works is because response.content yields HTML data from another view that can only be access if request.user.is_authenticated() is true. In other words, they must be logged in for response.content to yield the "logged in page". So given this, i can tell that the function obviously works for its practical purpose of logging the user in, however, i've been scouring the docs for hours trying to figure out why i can't access SESSION_KEY from the client session. All my reading suggests that the django test client is in fact stateful in nature and should store the session.
Can someone shed some light on this?
Ok after much searching and asking around on #django, i made a working solution for Django 1.6.x
from django.contrib.auth import SESSION_KEY, get_user_model
from django.test import Client
def setUp(self):
self.client = Client()
def test_login_view(self):
user_pk = get_user_model()._default_manager.get(username__exact='test_username_here').pk
response = self.client.post('/login/', {'username':'test_username_here', 'password':'test_password_here'})
self.assertEqual(self.client.session[SESSION_KEY], user_pk)
The test_login_view function will be the one evaluating the view in my app that handles user logins from the template form. First, i grab user_pk which is the real primary key of the given user in the database. I used get_user_model() instead of User.objects.get() because the former allows you to reference regardless of whether the User model is modified or not. Of course you can use the latter as well. Second, i go ahead and send the post request using the test client just like a standard user's browser would. Finally, i discovered that self.client.session[SESSION_KEY] contains the primary key of the logged in user. (If the login was successful, otherwise, it will simply yield a KeyError)
I'm writing a web app which has a page for admin tasks. One of the tasks is that the admin users must be able to edit other users details. Alas, I've fallen at quite a simple roadblock.
I've set up a very simple jQuery AJAX Get request, successfully transferring a string to the server and back. This is just background, but not the issue. The issue lies in retrieving other user's objects.
At the moment, with a username I know exists, this code which is accessed in views.py, produces a 500 Internal Server Error.
#login_required
def user_edit_getuser(request):
# Like before, get the request's context.
context = RequestContext(request)
inputname = request.GET['inputNameSend']
user_obj = User.objects.get(inputname)
return HttpResponse(inputname) #later will return a JSON String
get takes keyword arguments only: the key is the field to look up.
user_obj = User.objects.get(username=inputname)
Also, you should probably deal with the possibility that the GET request has no inputNameSend key.
For JS development, you can usually see the error page in the Chrome dev tools/Firebug console in the Network tab.