What I know:
We all know that flask has a useful query.get_or_404 we can call it to any class object and return the object or raise a 404 error if the object is not found.
The problem:
I have a very large application and using that function for my queries, but it becomes now a bit confusing when I'm sending data to the front end and other application consuming my APIs.
When an object they are querying is not found it returns 404 and the same behavior happens also when a page is not found.
What I want to achieve:
I would like to have a meaningful response that is different for every object that is not found, and different for the normal 404 error message.
Example :
if I have this query:
user = User.query.get_or_404(id)
if the user is not found I want to raise an HTTP error and return a message like user not found
What I have tried So far:
user = User.query.get(id)
if user:
#do something
else
return {'status':'01', 'description': 'user not found'}
# or raise a http error
The problem with this approach I can't maintain the application I'm working on, it will require me to change everywhere I was using get_or_404 with that code.
What I'm thinking about :
Creating a function like query.get_or_404but with another status message
like query.get_or_415 for example and add an error handler for 415 HTTP code so that if an object is not found it can return {'status':'0message:ge : 'object of the class is not found'}
How can I achieve it?
I have checked for that function in Flask but was unable to find it
Anyone with suggestions?
As noted by bereal, you should indeed use flask-SQLAlchemy's BaseQuery and add your own functionality from there. I'll check to see if I can add some messaging system to this approach as well.
import json
from flask import Flask, abort, Response
from flask_sqlalchemy import SQLAlchemy, BaseQuery
class CustomBaseQuery(BaseQuery):
def get_or_415(self, ident):
model_class_name = ''
try:
model_class_name = self._mapper_zero().class_.__name__
except Exception as e:
print(e)
rv = self.get(ident)
if not rv:
error_message = json.dumps({'message': model_class_name + ' ' + str(ident) + ' not found'})
abort(Response(error_message, 415))
return rv
app = Flask(__name__)
db = SQLAlchemy(app, query_class=CustomBaseQuery)
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True, nullable=False)
db.create_all()
user = User(username='foo')
db.session.add(user)
db.session.commit()
#app.route('/')
def index():
User.query.get_or_415(1)
User.query.get_or_415(2)
return ''
When going to the index, it returns:
{"message": "User 2 not found"}
Based on this answer
You can write like this:
from sqlalchemy.orm import Query
class MyQuery(Query):
def get_or_415(self, pk):
instance = self.get(pk)
if not instance:
raise HttpException(code=415)
return instance
Then you can use flask error handling to handle your exception and response the way you want
Related
I am writing APIs using stack FastAPI, Pydantic & SQL Alchemy and I have come across many cases where I had to query database to perform validations on payload values. Let's consider one example API, /forgot-password. This API will accept email in the payload and I need to validate the existence of the email in database. If the email exist in the database then necessary action like creating token and sending mail would be performed or else an error response against that field should be raise by Pydantic. The error responses must be the standard PydanticValueError response. This is because all the validation errors would have consistent responses as it becomes easy to handle for the consumers.
Payload -
{
"email": "example#gmail.com"
}
In Pydantic this schema and the validation for email is implemented as -
class ForgotPasswordRequestSchema(BaseModel):
email: EmailStr
#validator("email")
def validate_email(cls, v):
# this is the db query I want to perform but
# I do not have access to the active session of this request.
user = session.get(Users, email=v)
if not user:
raise ValueError("Email does not exist in the database.")
return v
Now this can be easily handled if the we simple create an Alchemy session in the pydantic model like this.
class ForgotPasswordRequestSchema(BaseModel):
email: EmailStr
_session = get_db() # this will simply return the session of database.
_user = None
#validator("email")
def validate_email(cls, v):
# Here I want to query on Users's model to see if the email exist in the
# database. If the email does. not exist then I would like to raise a custom
# python exception as shown below.
user = cls._session.get(Users, email=v) # Here I can use session as I have
# already initialised it as a class variable.
if not user:
cls.session.close()
raise ValueError("Email does not exist in the database.")
cls._user = user # this is because we want to use user object in the request
# function.
cls.session.close()
return v
But it is not a right approach as through out the request only one session should be used. As you can see in above example we are closing the session so we won't be able to use the user object in request function as user = payload._user. This means we will have to again query for the same row in request function. If we do not close the session then we are seeing alchemy exceptions like this - sqlalchemy.exc.PendingRollbackError.
Now, the best approach is to be able to use the same session in the Pydantic model which is created at the start of request and is also closing at the end of the request.
So, I am basically looking for a way to pass that session to Pydantic as context. Session to my request function is provided as dependency.
It is not recommended to query the database in pydantic schema. Instead use session as a dependency.
If you want to raise errors like pydantic validation error you might need this:
def raise_custom_error(exc: Exception, loc: str, model: BaseModel, status_code=int, **kwargs):
"""
This method will return error responses using pydantic error wrapper (similar to pydantic validation error).
"""
raise HTTPException(
detail=json.loads(ValidationError([ErrorWrapper(exc(**kwargs), loc=loc)], model=model).json()),
status_code=status_code,
)
Usage
class PayloadSchema(BaseModel):
email: EmailStr
#app_router.post('/forgot-password')
def forgot_password(
payload: PayloadSchema,
session: Session = Depends(get_db),
background_tasks: BackgroundTasks
):
existing_user = db.get(Users, email=payload.email)
if(existing_user):
raise_custom_error(
PydanticValueError, "email", PayloadSchema, status.HTTP_400_BAD_REQUEST
)
background_tasks(send_email, email=payload.email)
Don't do that!
The purpose of pydantic classes is to store dictionaries in a legit way, as they have IDE support and are less error prone. The validators are there for very simple stuff that doesn't touch other parts of system (like is integer positive or does email satisfy the regex).
Saying that, you should use the dependencies. That way you can be sure you have single session during processing all request and because of context manager the session will be closed in any case.
Final solution could look like this:
from fastapi import Body, Depends
from fastapi.exceptions import HTTPException
def get_db():
db = your_session_maker
try:
yield db
finally:
db.close()
#app.post("/forgot-password/")
def forgot_password(email: str = Body(...), db: Session = Depends(get_db)):
user = db.get(Users, email=email)
if not user:
# If you really need to, you can for some reason raise pydantic exception here
raise HTTPException(status_code=400, detail="No email")
When I try getting a kind from my datastore it returns NoneType, as if the query is empty. I know the datastore is working properly while saving, but pulling a kind from the a query is not.
Also using the GQL Query in the Google cloud Console website and using SELECT * FROM User does return all the kinds. User kind has no parents, it is at the root. I made sure all the properties are indexed as well.
I am not sure what I am doing wrong on GET.
MyApp.py
import webapp2
from google.appengine.ext import ndb
from google.appengine.ext.db import GqlQuery
class MainHandler(webapp2.RequestHandler):
def post(self):
message = self.request.body
message = message.splitlines()
if message[0] == "register":
user = User.create_user(message[1], message[2], message[3])
user_key = User.save_user(user)
if user_key is not None:
self.response.write(user_key)
else:
user = User.get_by_id(User.email == message[0])
if User.token == message[1]:
self.response.write("CURRENT")
else:
User.token = message[1]
User.save_user(user)
self.response.write("UPDATED")
def get(self):
self.response.write("CONNECTED")
user= User.query().get()
self.response.write("\n" + query.email)
class User(ndb.Model):
email = ndb.StringProperty()
token = ndb.StringProperty()
name = ndb.StringProperty()
#classmethod
def create_user(cls, email, token, name):
user = User(email=email, token=token, name=name, id=email)
return user
#classmethod
def save_user(cls, user):
user_key = user.put()
return user_key
#classmethod
def get_user(cls, email):
return User.get_by_id(User.email == email)
app = webapp2.WSGIApplication([
('/', MainHandler)
], debug=True)
You seem to be confusing .get_by_id() with a query.
The get_by_id method is actually mapped to ndb.Model._get_by_id which invokes ndb.Model._get_by_id_async, which requires an entity key identifier to determine the entity's key used to do a direct entity lookup (not a query!). From appengine.ext.ndb.model.py:
#classmethod
#utils.positional(3)
def _get_by_id_async(cls, id, parent=None, app=None, namespace=None,
**ctx_options):
"""Returns an instance of Model class by ID (and app, namespace).
This is the asynchronous version of Model._get_by_id().
"""
key = Key(cls._get_kind(), id, parent=parent, app=app, namespace=namespace)
return key.get_async(**ctx_options)
But in your code you're passing as id a bool: User.email == message[0], which most likely won't match any existing entity key identifiers, hence the None result causing the error you see.
Since the info you have available is the value of an entity's property (the email value) you probably want to perform a query, something along these lines:
results = User.query(User.email == message[0]).fetch(limit=1)
if results:
user = results[0]
So I figured out what was wrong. So there seems to be an issue with the way Google Cloud SDK is set up on my computer. When running the same code on the google servers rather than on my network everything seems to work properly.
I'm working with Miguel Grinberg's Flask REST API repo, and I'm failing to return JSON paginated results. The examples online use html templates, but I just want to return a number of results (20), and eventually return links for the previous and next pages. When I return the code immediately following this sentence, I get "pagination object is not iterable":
def get_customers():
return jsonify({'customers': [customer.get_url() for customer in
Customer.query.paginate(page=1, per_page=1)]})
I understand I'm passing the wrong object, but I'm not sure if I should use another module, or if I'm on the right path. Does anyone have a suggestion to reach my end goal?
The original code in Miguel's repo is:
#app.route('/customers/', methods=['GET'])
def get_customers():
return jsonify({'customers': [customer.get_url() for customer in
Customer.query.all()]})
The whole file is here: https://github.com/miguelgrinberg/oreilly-flask-apis-video/blob/a460ad9df2e58c13b90f183e81b4e8953eb186cb/orders/api.py
The relevant code I'm working with:
class Customer(db.Model):
__tablename__ = 'customers'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(64), index=True)
def get_url(self):
return url_for('get_customer', id=self.id, _external=True)
def export_data(self):
return {
'self_url': self.get_url(),
'name': self.name
}
def import_data(self, data):
try:
self.name = data['name']
except KeyError as e:
raise ValidationError('Invalid customer: missing ' + e.args[0])
return self
#app.route('/customers/', methods=['GET'])
def get_customers():
return jsonify({'customers': [customer.get_url() for customer in
Customer.query.paginate(page=1, per_page=1)]})
#app.route('/customers/<int:id>', methods=['GET'])
def get_customer(id):
return jsonify(Customer.query.get_or_404(id).export_data())
See the API docs.
If you want to iterate over a Pagination object, use (for example)
Customer.query.paginate(page=1, per_page=1).items
which is a collection of the items for that page.
I'm aware there are many other questions about the exact same issue, but I've tried their answers and none have worked so far.
I'm trying to delete records from a table which has relationships with other tables. The foreign key in those tables are nullable=false, so trying to delete a record which is in use by another table should raise an exception.
But even when surrounding the delete statement with a catchall try-except the error is still not caught, so I suspect the exception may be raised somewhere else.
I'm using SQLite with SQLAlchemy in a Pyramid framework, and my session is configured with the ZopeTransactionExtension.
This is how I'm trying to delete:
In views.py
from sqlalchemy.exc import IntegrityError
from project.app.models import (
DBSession,
foo)
#view_config(route_name='fooview', renderer='json', permission='view')
def fooview(request):
""" The fooview handles different cases for foo
depending on the http method
"""
if request.method == 'DELETE':
if not request.has_permission('edit'):
return HTTPForbidden()
deleteid = request.matchdict['id']
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
try:
qry = DBSession.delete(deletethis)
transaction.commit()
if qry == 0:
return HTTPNotFound(text=u'Foo not found')
except IntegrityError:
DBSession.rollback()
return HTTPConflict(text=u'Foo in use')
return HTTPOk()
In models.py I set up DBSession and my models:
from zope.sqlalchemy import ZopeTransactionExtension
from sqlalchemy.orm import (
scoped_session,
sessionmaker,
relationship,
backref,
)
DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension('changed')))
Base = declarative_base()
class foo(Base):
""" foo defines a unit used by bar
"""
__tablename__ = 'foo'
id = Column(Integer, primary_key=True)
name = Column(Text(50))
bars = relationship('bar')
class bar(Base):
__tablename__ = 'bar'
id = Column(Integer, primary_key=True)
fooId = Column(Integer, ForeignKey('foo.id'), nullable=False)
foo = relationship('foo')
And in __init__.py I configure my session like so:
from project.app.models import (
DBSession,
Base,
)
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
engine = engine_from_config(settings, 'sqlalchemy.')
# fix for association_table cascade delete issues
engine.dialect.supports_sane_rowcount = engine.dialect.supports_sane_multi_rowcount = False
DBSession.configure(bind=engine)
Base.metadata.bind = engine
Using this setup I get
IntegrityError: (IntegrityError) NOT NULL constraint failed
Traceback here.
If I replace transaction.commit() with DBSession.flush(), I get
ResourceClosedError: This transaction is closed
And if I remove the transaction.commit(), I still get the same error, but without a clear point of origin.
UPDATE:
I ran some nose tests, and in some cases, but not all, the exception was handled correctly.
In my tests I import the session and configure it:
from optimate.app.models import (
DBSession,
Base,
foo)
def _initTestingDB():
""" Build a database with default data
"""
engine = create_engine('sqlite://')
Base.metadata.create_all(engine)
DBSession.configure(bind=engine)
with transaction.manager:
# add test data
class TestFoo(unittest.TestCase):
def setUp(self):
self.config = testing.setUp()
self.session = _initTestingDB()
def tearDown(self):
DBSession.remove()
testing.tearDown()
def _callFUT(self, request):
from project.app.views import fooview
return fooview(request)
def test_delete_foo_keep(self):
request = testing.DummyRequest()
request.method = 'DELETE'
request.matchdict['id'] = 1
response = self._callFUT(request)
# foo is used so it is not deleted
self.assertEqual(response.code, 409)
def test_delete_foo_remove(self):
_registerRoutes(self.config)
request = testing.DummyRequest()
request.method = 'DELETE'
request.matchdict['id'] = 2
response = self._callFUT(request)
# foo is not used so it is deleted
self.assertEqual(response.code, 200)
Does anyone know what's going on?
Might be you just "doing it wrong". Your question addresses two issues. Handling transaction level errors raised by database integrity errors and modelling application code/models/queries to implement business logic. My answer focuses on writing code that fits common patterns while using pyramid_tm for transaction management and sqlalchemy as an ORM.
In Pyramid, if you've configured your session (which the scaffold does for you automatically) to use the ZopeTransactionExtension, then session is not flushed/committed until after the view has executed. If you want to catch any SQL errors yourself in your view, you need to force a flush to send the SQL to the engine. DBSession.flush() should do it after the delete(...).
If you raise any of 4xx/5xx HTTP return codes like pyramid exception HTTPConflict the transaction will be aborted.
#view_config(route_name='fooview', renderer='json', permission='view')
def fooview(request):
""" The fooview handles different cases for foo
depending on the http method
"""
if request.method == 'DELETE':
if not request.has_permission('edit'):
return HTTPForbidden()
deleteid = request.matchdict['id']
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
if not deletethis:
raise HTTPNotFound()
try:
DBSession.delete(deletethis)
DBSession.flush()
except IntegrityError as e:
log.debug("delete operation not possible for id {0}".format(deleteid)
raise HTTPConflict(text=u'Foo in use')
return HTTPOk()
This excerpt from todopyramid/models.py highlights how to delete a collection item without using DBSession object.
def delete_todo(self, todo_id):
"""given a todo ID we delete it is contained in user todos
delete from a collection
http://docs.sqlalchemy.org/en/latest/orm/session.html#deleting-from-collections
https://stackoverflow.com/questions/10378468/deleting-an-object-from-collection-in-sqlalchemy"""
todo_item = self.todo_list.filter(
TodoItem.id == todo_id)
todo_item.delete()
This sample code from pyramid_blogr show clearly how simple pyramid view code to delete SQL database items could look like. Usually you do not have to interact with the transaction. This is a feature - as advertised as one the unique feature of pyramid. Just pick any of the available pyramid tutorials that use sqlalchemy and try to stick to the patterns as much as possible. If you address the problem at the application model level the transaction machinery will hide in the background unless you have a clear need for its services.
#view_config(route_name='blog_action', match_param="action=delete", permission='delete')
def blog_delete(request):
entry_id = request.params.get('id', -1)
entry = Entry.by_id(entry_id)
if not entry:
return HTTPNotFound()
DBSession.delete(entry)
return HTTPFound(location=request.route_url('home'))
To provide meaningful error messages to application users you either catch errors on database contraints at database model layer or at pyramid view layer. Catching sqlalchemy exceptions to provide error messages could look like in this sample code
from sqlalchemy.exc import OperationalError as SqlAlchemyOperationalError
#view_config(context=SqlAlchemyOperationalError)
def failed_sqlalchemy(exception, request):
"""catch missing database, logout and redirect to homepage, add flash message with error
implementation inspired by pylons group message
https://groups.google.com/d/msg/pylons-discuss/BUtbPrXizP4/0JhqB2MuoL4J
"""
msg = 'There was an error connecting to database'
request.session.flash(msg, queue='error')
headers = forget(request)
# Send the user back home, everything else is protected
return HTTPFound(request.route_url('home'), headers=headers)
References
Trying to catch integrity error with SQLAlchemy
pyramid_tm Usage
What the Zope Transaction Manager Means To Me (and you)
Not sure if this helps - I did not quite capture from the traceback what goes wrong, would need more time. But you can use transaction manager like this:
from sqlalchemy.exc import IntegrityError
try:
with transaction.manager:
deletethis = DBSession.query(foo).filter_by(id=deleteid).first()
qry = DBSession.delete(deletethis)
if qry == 0:
return HTTPNotFound()
# transaction.manager commits when with context manager exits here
except IntegrityError:
DBSession.rollback()
return HTTPConflict()
return HTTPOk()
I am attempting to render good ol' JSON via return {'category_name': category.category_name}in a views.py file. I am using Pyramid Web Framework. I am using a custom RESTful web api and a CRUD design in the SQLALCHEMY database.
For some reason, I keep getting the Traceback error:
views/views.py", line 112, in get_category
return {'category': category.category_name}
AttributeError: 'list' object has no attribute 'category_name'
All I want to do is print this out to HTML or render it to HTML so that I can see what was created. Category does have the attribute category_name...which is confusing. Perhaps to return a list value, I have to use special syntax? I couldn't find anything online that made sense to me (this was the most relevant), but any guidance would be greatly appreciated! I am sure it is something simple.
GET method: site of the return object (meant to render on Jinja2 templates later):
#view_config(route_name='category', request_method='GET', renderer='json')
def get_category(request):
with transaction.manager:
category_id = int(request.matchdict['id'])
category = api.retrieve_category(category_id)
if category is None:
raise HTTPNotFound()
return {'category_name': category.category_name}
This is what the POST method looks like (very much like the Create function in the DB api):
#view_config(route_name='categories', request_method='POST', renderer='json')
def post_category(request):
with transaction.manager:
category_name = request.params['category_name']
category = api.create_category(category_name)
return HTTPCreated(location=request.route_url('category',id=id))
DB API for creating a category (this has a Many-to-Many relationship with Assessment, thus the list):
def create_category(self, category_name):
new_category = Category(category_name)
self.session.add(new_category)
print(new_category)
self.session.commit()
Retrieve method:
def retrieve_category(self, something_unique):
if isinstance(something_unique, int):
return self.session.query(Category).\
filter(Category.category_id == something_unique).all() # multiple categories
elif isinstance(something_unique, basestring):
print(something_unique) # added
return self.session.query(Category).\
filter(Category.category_name == something_unique).one()
print(something_unique)
if NoResultFound:
raise NotFoundError('No results found')
elif MultipleResultsFound:
raise MultipleResultsFound('Too many results found')
elif isinstance(something_unique, Category):
return something_unique
else:
raise ValueError('Value being passed is an object')
convenience api (inheritance):
def create_assessment(self, name, text, username, videoname, category_names):
user = self.retrieve_user(username)
video = self.retrieve_video(videoname)
cat_objects = [self.retrieve_category(category_name) for category_name in category_names]
return super(ConvenienceAPI, self).create_assessment(name, text, user, video, cat_objects)
api.retrieve_category() is returning a list, as the exception shows. What are you expecting it to return?
You need to check what your API is returning. Possibly you could access the first item in the returned list so this might work:
return {'category_name': category[0].category_name}
However, if the API is actually returning an empty list ([]) to signify no result, then you need to check for that rather than for None:
if not category:
raise HTTPNotFound()