I'm working with Flask-Marshmallow for validating request and response schemas in Flask app. I was able to do simple validations for request.form and request.args when there are simple fields like Int, Str, Float etc.
I have a case where I need to upload a file using a form field - file_field. It should contain the file content.
How can I validate if this field is present or not and what is the format of file etc.
Is there any such field in Marshmallow that I can use like fields.Int() or fields.Str()
I have gone through the documentation here but haven't found any such field.
You can use fields.Raw:
import marshmallow
class CustomSchema(marshmallow.Schema):
file = marshmallow.fields.Raw(type='file')
If you are using Swagger, you would then see something like this:
Then in your view you can access the file content with flask.request.files.
For a full example and more advanced topics, check out my project.
You can use either Field or Raw field to represent a general field, then set the correct OpenAPI properties for this field (i.e. type and format).
In OpenAPI 3, you should set type to string instead of file, then set format to binary or byte based on your file content:
from marshmallow import Schema, fields
class MySchema(Schema):
file = fields.Raw(metadata={'type': 'string', 'format': 'binary'})
See the docs for more details.
By the way, from marshmallow 3.10.0, using keyword arguments to pass OpenAPI properties (description, type, example etc.) was deprecated and will be removed in marshmallow 4, use the metadata dict to pass them instead.
As of 7 December, 2022, marshmallow v3.19.0 doesn't work with Raw(metadata={}) or Raw(type='binary/file'). Hence I tried Field() and it works fine.
To validate JPG, PNG files, I do that while #validates_schema by catching it's file type. I have also saved the file after in #post_load method.
class MySchema(Schema):
icon_url = Field(metadata={'type': 'string', 'format': 'byte'}, allow_none=True)
#validates_schema
def validate_uploaded_file(self, in_data, **kwargs):
errors = {}
file: FileStorage = in_data.get("icon_url", None)
if file is None:
# if any file is not uploaded, skip validation
pass
elif type(file) != FileStorage:
errors["icon_url"] = [
f"Invalid content. Only PNG, JPG/JPEG files accepted"]
raise ValidationError(errors)
elif file.content_type not in {"image/jpeg", "image/png"}:
errors["icon_url"] = [
f"Invalid file_type: {file.content_type}. Only PNG, JPG/JPEG images accepted."]
raise ValidationError(errors)
return in_data
#post_load
def post_load(self, loaded_obj, **kwargs):
if loaded_obj.icon_url:
sec_filename = secure_filename(
f'{loaded_obj.name}.{loaded_obj.icon_url.filename.split(".")[-1]}')
loaded_obj.icon_url.save(
f"{current_app.config['PUBLIC_IMAGES_FOLDER']}{sec_filename}")
loaded_obj.icon_url = f'{current_app.config["PUBLIC_IMAGES_URL"]}{sec_filename}'
return loaded_obj
Related
I have a use case where I'm attempting to override an Image URL if it exists in our database.
Here is the section of the queryset that is grabbing the ImageField from via an F() query.
preferred_profile_photo=Case(
When(
# agent not exists
Q(agent__id__isnull=False),
then=F("agent__profile__profile_photo"),
),
default=Value(None, output_field=ImageField()),
)
The Case-When is resolving correctly, but the issue is the value returns from F("agent__profile__profile_photo") is not the URL than can be used by the UI. Instead it is something like:
"agentphoto/09bd7dc0-62f6-49ab-blah-6c57b23029d7/profile/1665342685--77e51d9c5asdf345364f774d0b2def48.jpeg"
Typically, I'd retrieve the URL via agent.profile.profile_photo.url, but I receive the following when attempting to perform preferred_profile_photo.url:
AttributeError: 'str' object has no attribute 'url'.
I've tried wrapping in Value(..., output_field=ImageField()) with no luck.
The crux here is retrieving the url from the ImageField after resolving from F()
For reference, I'm using storages.backends.s3boto3.S3Boto3Storage.
I was able to implement this using some information from this post.
Here is the helper function I used to implement:
from django.core.files.storage import get_storage_class
def get_media_storage_url(file_name: str) -> str:
"""Takes the postgres stored ImageField value and converts it to
the proper S3 backend URL. For use cases, where calling .url on the
photo field is not feasible.
Args:
file_name (str): the value of the ImageField
Returns:
str: the full URL of the object usable by the UI
"""
media_storage = get_storage_class()()
return media_storage.url(name=file_name)
Django doesn't store full URL in DB. It builds absolute url on code level. Quote from docs:
All that will be stored in your database is a path to the file (relative to MEDIA_ROOT).
You can use build_absolute_uri() method to get full URL of your files. For example you can do it on serializer level:
class YourSerializer(ModelSerializer):
preferred_profile_photo = serializers.SerializerMethodField()
class Meta:
model = YourModel
fields = [
'preferred_profile_photo',
]
def get_preferred_profile_photo(self, obj):
request = self.context.get('request')
return request.build_absolute_uri(obj.preferred_profile_photo)
I am using python rebar for validating request_body_schema, it works well. we can validate input body arguments. Its cool, we do not want to implement any manual input validations like adding if statements.
Same way I could not able to validate response arguments.
But flask_rebar mentioned we can implement Link
Opting In to Response Validation
There are two ways to opt-in to response validation:
Globally, via validate_on_dump attribute of your Rebar instance. Using this method, it is easy to turn on validation for things like test cases, while reaping performance gains by leaving it off in your production endpoints (assuming your API contract testing is sufficient to guarantee that your API can’t return invalid data).
At schema level, via flask_rebar.validation.RequireOnDumpMixin (including if you use our legacy pre-canned ResponseSchema as the base class for your schemas). Any schema that includes that mixin is automatically opted in to response validation, regardless of global setting. Note that in Flask-Rebar 2, that mixin serves only as a “marker” to trigger validation; we plan to augment/replace this with ability to use SchemaOpts as a more logical way of accomplishing the same thing in the near future (https://github.com/plangrid/flask-rebar/issues/252).
But I am not getting any example, Can any body help me with example
my code:
from marshmallow import fields, Schema
from flask import Flask
from flask_rebar import Rebar, RequestSchema, get_validated_body
class CreateAccountSchema(RequestSchema):
email = fields.String(required=True)
country = fields.String(required=True)
default_currency = fields.String(required=True)
class AccountSchema(Schema):
id = fields.String()
email = fields.String()
country = fields.String()
default_currency = fields.String(required=True) # if this is not passed raise error
rebar = Rebar()
registry = rebar.create_handler_registry(prefix="/v1")
#registry.handles(
rule='/accounts',
method='POST',
marshal_schema={201: AccountSchema()},
request_body_schema=CreateAccountSchema(),)
def get_todos():
"""
This docstring will be rendered as the operation's description in
the auto-generated OpenAPI specification.
"""
body = get_validated_body()
body = rebar.validated_body
data = {"id": "myname", "email": "myemail", "country": "any"}
return data, 201
#registry.handles(
rule='/values',
method='GET',
marshal_schema=None,)
def get_values():
"""
This docstring will be rendered as the operation's description in
the auto-generated OpenAPI specification.
"""
data = {"id": "myname", "email": "myemail", "country": "any"}
return 'Hello, Poorvika'
def create_app(name) -> Flask:
app = Flask(name)
rebar.init_app(app)
return app
if __name__ == '__main__':
create_app(__name__).run()
To get the example to work in Flask-Rebar 2.0, you have to replace marshal_schema argument in the #registry.handle decorator with response_body_schema. marshal_schema is the old deprecated name which was removed in Rebar 2.0. The new name response_body_schema was introduced in Rebar 1.7.
response_body_schema is a Marshmallow schema that will be used marshal the return value of the function. marshmallow.Schema.dump will be called on the return value. response_body_schema can also be a dictionary mapping status codes to Marshmallow schemas - see Marshaling. NOTE: In Flask-Rebar 1.0-1.7.0, this was referred to as marshal_schema. It is being renamed and both names will function until version 2.0.
– Basics - Flask-Rebar documentation
https://flask-restplus.readthedocs.io/en/stable/marshalling.html
`#api.marshal_with`
user reponse marshaling
I'm trying to set up a Webservice that accepts Images via POST methods.
I'm using Python Flask as well as Flask-Apispec to create a Swagger documentation. I thus include this Marshmallow Schema to define which parameters are accepted:
class UploadRequestSchema(Schema):
image = fields.Raw(type="file")
I now also want to document that only png-images are accepted and validate this to be true in Marshmallow.
Because of this, I've tried setting up a validator
class FileExtension(Validator)
def __call__(self, value, **kwargs):
print(value)
print(type(value))
for key in kwargs:
print(key)
//if filename ends in ".png"
return true
class UploadRequestSchema(Schema):
image = fields.Raw(type="file", validate=FileExtension())
However, the console output for this code is simply
[object File]
<class 'str'>
So value is simply a String with content "[object File]" and kwargs is empty. How can I access the file submitted to check its name? Alternatively, in what other way can I validate file uploads in Marshmallow?
The value should be an instance of Werkzueg's FileStorage object, you can validate against its filename or mimetype attributes.
from marshmallow import ValidationError
class FileExtension(Validator)
def __call__(self, value, **kwargs):
if not value.filename.endswith('.png'):
raise ValidationError('PNG only!')
return true
flask_restful.reqparse has been deprecated (https://flask-restful.readthedocs.io/en/latest/reqparse.html):
The whole request parser part of Flask-RESTful is slated for removal and will be replaced by documentation on how to integrate with other packages that do the input/output stuff better (such as marshmallow). This means that it will be maintained until 2.0 but consider it deprecated. Don’t worry, if you have code using that now and wish to continue doing so, it’s not going to go away any time too soon.
I've looked briefly at Marshmallow and still a bit confused about how to use it if I wanted to replace reqparse.RequestParser(). What would we write instead of something like the following:
from flask import Flask, request, Response
from flask_restful import reqparse
#app.route('/', methods=['GET'])
def my_api() -> Response:
parser = reqparse.RequestParser()
parser.add_argument('username', type=str, required=True)
args = parser.parse_args()
return {'message': 'cool'}, 200
(after half an hour of reading some more documentation…)
RequestParser looks at the MultiDict request.values by default (apparently query parameters, then form body parameters according to https://stackoverflow.com/a/16664376/5139284). So then we just need to validate the data in request.values somehow.
Here's a snippet of some relevant code from Marshmallow. It seems a good deal more involved than reqparse: first you create a schema class, then instantiate it, then have it load the request JSON. I'd rather not have to write a separate class for each API endpoint. Is there something more lightweight similar to reqparse, where you can write all the types of the argument validation information within the function defining your endpoint?
from flask import Flask, request, Response
from flask_restful import reqparse
from marshmallow import (
Schema,
fields,
validate,
pre_load,
post_dump,
post_load,
ValidationError,
)
class UserSchema(Schema):
id = fields.Int(dump_only=True)
email = fields.Str(
required=True, validate=validate.Email(error="Not a valid email address")
)
password = fields.Str(
required=True, validate=[validate.Length(min=6, max=36)], load_only=True
)
joined_on = fields.DateTime(dump_only=True)
user_schema = UserSchema()
#app.route("/register", methods=["POST"])
def register():
json_input = request.get_json()
try:
data = user_schema.load(json_input)
except ValidationError as err:
return {"errors": err.messages}, 422
# etc.
If your endpoints share any commonalities in schema, you can use fields.Nested() to nest definitions within each Marshmallow class, which may save on code writing for each endpoint. Docs are here.
For example, for operations that update a resource called 'User', you would likely need a standardised subset of user information to conduct the operation, such as user_id, user_login_status, user_authorisation_level etc. These can be created once and nested in new classes for more specific user operations, for example updating a user's account:
class UserData(Schema):
user_id = fields.Int(required=True)
user_login_status = fields.Boolean(required=True)
user_authentication_level = fields.Int(required=True)
# etc ....
class UserAccountUpdate(Schema):
created_date = fields.DateTime(required=True)
user_data = fields.Nested(UserData)
# account update fields...
I have a model for which I am making an api in tastypie. I have a field which stores the path to a file which I maintain manually (I am not using FileField since users are not uploading the files). Here is a gist of a model:
class FooModel(models.Model):
path = models.CharField(max_length=255, null=True)
...
def getAbsPath(self):
"""
returns the absolute path to a file stored at location self.path
"""
...
Here is my tastypie config:
class FooModelResource(ModelResource):
file = fields.FileField()
class Meta:
queryset = FooModel.objects.all()
def dehydrate_file(self, bundle):
from django.core.files import File
path = bundle.obj.getAbsPath()
return File(open(path, 'rb'))
In the api in the file field this returns full path to a file. I want tastypie to be able to serve the actual file or at least an url to a file. How do I do that? Any code snippets are appreciated.
Thank you
Decide on a URL scheme how your files will be exposed through the APIs first. You don't really need file or dehydrate_file (unless you want to change the representation of the file for the model itself in Tastypie). Instead just add an additional action on the ModelResource. Example:
class FooModelResource(ModelResource):
file = fields.FileField()
class Meta:
queryset = FooModel.objects.all()
def override_urls(self):
return [
url(r"^(?P<resource_name>%s)/(?P<pk>\w[\w/-]*)/download%s$" % (self._meta.resource_name, trailing_slash()), self.wrap_view('download_detail'), name="api_download_detail"),
]
def download_detail(self, request, **kwargs):
"""
Send a file through TastyPie without loading the whole file into
memory at once. The FileWrapper will turn the file object into an
iterator for chunks of 8KB.
No need to build a bundle here only to return a file, lets look into the DB directly
"""
filename = self._meta.queryset.get(pk=kwargs[pk]).file
wrapper = FileWrapper(file(filename))
response = HttpResponse(wrapper, content_type='text/plain') #or whatever type you want there
response['Content-Length'] = os.path.getsize(filename)
return response
GET .../api/foomodel/3/
Returns:
{
...
'file' : 'localpath/filename.ext',
...
}
GET .../api/foomodel/3/download/
Returns:
...contents of actual file...
Alternatively you could create a non-ORM Sub Resource file in FooModel. You would have to define resource_uri (how to uniquely identify each instance of the resource), and override dispatch_detail to do exactly what download_detail above does.
The only conversion tastypie does on a FileField is to look for an 'url' attribute on what you return, and return that if it exists, else it will return the string-ized object, which as you have noticed is just the filename.
If you want to return the file content as a field, you will need to handle the encoding of the file. You have a few options:
Simplest: Use CharField and use the base64 module to convert the bytes read from the file into a string
More general but functionally equivalent: write a custom tastypie Serializer that knows how to turn File objects into string representations of their contents
Override the get_detail function of your resource to serve just the file using whatever content-type is appropriate, to avoid the JSON/XML serialization overhead.