I have a model for which I am making an api in tastypie. I have a field which stores the path to a file which I maintain manually (I am not using FileField since users are not uploading the files). Here is a gist of a model:
class FooModel(models.Model):
path = models.CharField(max_length=255, null=True)
...
def getAbsPath(self):
"""
returns the absolute path to a file stored at location self.path
"""
...
Here is my tastypie config:
class FooModelResource(ModelResource):
file = fields.FileField()
class Meta:
queryset = FooModel.objects.all()
def dehydrate_file(self, bundle):
from django.core.files import File
path = bundle.obj.getAbsPath()
return File(open(path, 'rb'))
In the api in the file field this returns full path to a file. I want tastypie to be able to serve the actual file or at least an url to a file. How do I do that? Any code snippets are appreciated.
Thank you
Decide on a URL scheme how your files will be exposed through the APIs first. You don't really need file or dehydrate_file (unless you want to change the representation of the file for the model itself in Tastypie). Instead just add an additional action on the ModelResource. Example:
class FooModelResource(ModelResource):
file = fields.FileField()
class Meta:
queryset = FooModel.objects.all()
def override_urls(self):
return [
url(r"^(?P<resource_name>%s)/(?P<pk>\w[\w/-]*)/download%s$" % (self._meta.resource_name, trailing_slash()), self.wrap_view('download_detail'), name="api_download_detail"),
]
def download_detail(self, request, **kwargs):
"""
Send a file through TastyPie without loading the whole file into
memory at once. The FileWrapper will turn the file object into an
iterator for chunks of 8KB.
No need to build a bundle here only to return a file, lets look into the DB directly
"""
filename = self._meta.queryset.get(pk=kwargs[pk]).file
wrapper = FileWrapper(file(filename))
response = HttpResponse(wrapper, content_type='text/plain') #or whatever type you want there
response['Content-Length'] = os.path.getsize(filename)
return response
GET .../api/foomodel/3/
Returns:
{
...
'file' : 'localpath/filename.ext',
...
}
GET .../api/foomodel/3/download/
Returns:
...contents of actual file...
Alternatively you could create a non-ORM Sub Resource file in FooModel. You would have to define resource_uri (how to uniquely identify each instance of the resource), and override dispatch_detail to do exactly what download_detail above does.
The only conversion tastypie does on a FileField is to look for an 'url' attribute on what you return, and return that if it exists, else it will return the string-ized object, which as you have noticed is just the filename.
If you want to return the file content as a field, you will need to handle the encoding of the file. You have a few options:
Simplest: Use CharField and use the base64 module to convert the bytes read from the file into a string
More general but functionally equivalent: write a custom tastypie Serializer that knows how to turn File objects into string representations of their contents
Override the get_detail function of your resource to serve just the file using whatever content-type is appropriate, to avoid the JSON/XML serialization overhead.
Related
I have a use case where I'm attempting to override an Image URL if it exists in our database.
Here is the section of the queryset that is grabbing the ImageField from via an F() query.
preferred_profile_photo=Case(
When(
# agent not exists
Q(agent__id__isnull=False),
then=F("agent__profile__profile_photo"),
),
default=Value(None, output_field=ImageField()),
)
The Case-When is resolving correctly, but the issue is the value returns from F("agent__profile__profile_photo") is not the URL than can be used by the UI. Instead it is something like:
"agentphoto/09bd7dc0-62f6-49ab-blah-6c57b23029d7/profile/1665342685--77e51d9c5asdf345364f774d0b2def48.jpeg"
Typically, I'd retrieve the URL via agent.profile.profile_photo.url, but I receive the following when attempting to perform preferred_profile_photo.url:
AttributeError: 'str' object has no attribute 'url'.
I've tried wrapping in Value(..., output_field=ImageField()) with no luck.
The crux here is retrieving the url from the ImageField after resolving from F()
For reference, I'm using storages.backends.s3boto3.S3Boto3Storage.
I was able to implement this using some information from this post.
Here is the helper function I used to implement:
from django.core.files.storage import get_storage_class
def get_media_storage_url(file_name: str) -> str:
"""Takes the postgres stored ImageField value and converts it to
the proper S3 backend URL. For use cases, where calling .url on the
photo field is not feasible.
Args:
file_name (str): the value of the ImageField
Returns:
str: the full URL of the object usable by the UI
"""
media_storage = get_storage_class()()
return media_storage.url(name=file_name)
Django doesn't store full URL in DB. It builds absolute url on code level. Quote from docs:
All that will be stored in your database is a path to the file (relative to MEDIA_ROOT).
You can use build_absolute_uri() method to get full URL of your files. For example you can do it on serializer level:
class YourSerializer(ModelSerializer):
preferred_profile_photo = serializers.SerializerMethodField()
class Meta:
model = YourModel
fields = [
'preferred_profile_photo',
]
def get_preferred_profile_photo(self, obj):
request = self.context.get('request')
return request.build_absolute_uri(obj.preferred_profile_photo)
What is the best way the update the total field with value total the rows the file?
Implement in model or views or other? How to make The file registration will always be through django-admin
models.py
class Registry(models.Model):
file_upload = models.FileField(blank=True, null=False) #csv or xlsx
total = models.CharField(max_length=100, null=True, blank=True, default=None)
def save(self):
with open(self.file_upload) as f:
self.total = sum(1 for line in f)
return self.total
Error:
TypeError: expected str, bytes or os.PathLike object, not FieldFile
You can simply read the file content of the uploaded file as using .read() method.
And then do whatever you want to do with that content.
def save(self):
self.total = sum(1 for line in self.file_upload.read())
super(Registry, self).save(*args, **kwargs)
No need to again open at OS level.
The output of self.file_upload is a FieldFile object. You should change it to self.file_upload.path where will give you the string path of file.
And to makesure your self.file_upload is not None/Null, you should validate it also.
def save(self):
if self.file_upload:
with open(self.file_upload.path) as f:
....
You can read this docs for more https://docs.djangoproject.com/en/dev/topics/files/#using-files-in-models
I generally choose model part if I need to use the method for most of the instances that will be created. But in this case, I probably choose Django Forms to handle this business logic. By the way, you can choose all the possibilities. At least you can achieve what you need in both cases. If the business logics changes very often, I can suggest to move these logics to views or forms.
The error you have encountered is about open statement, which requires one of the types that declared in error message. To achieve that, you can change self.file_upload to self.file_upload.path which is the path that file uploaded. I strongly recommend you to use csv module or an excel file library to handle file read operations.
I'm working with Flask-Marshmallow for validating request and response schemas in Flask app. I was able to do simple validations for request.form and request.args when there are simple fields like Int, Str, Float etc.
I have a case where I need to upload a file using a form field - file_field. It should contain the file content.
How can I validate if this field is present or not and what is the format of file etc.
Is there any such field in Marshmallow that I can use like fields.Int() or fields.Str()
I have gone through the documentation here but haven't found any such field.
You can use fields.Raw:
import marshmallow
class CustomSchema(marshmallow.Schema):
file = marshmallow.fields.Raw(type='file')
If you are using Swagger, you would then see something like this:
Then in your view you can access the file content with flask.request.files.
For a full example and more advanced topics, check out my project.
You can use either Field or Raw field to represent a general field, then set the correct OpenAPI properties for this field (i.e. type and format).
In OpenAPI 3, you should set type to string instead of file, then set format to binary or byte based on your file content:
from marshmallow import Schema, fields
class MySchema(Schema):
file = fields.Raw(metadata={'type': 'string', 'format': 'binary'})
See the docs for more details.
By the way, from marshmallow 3.10.0, using keyword arguments to pass OpenAPI properties (description, type, example etc.) was deprecated and will be removed in marshmallow 4, use the metadata dict to pass them instead.
As of 7 December, 2022, marshmallow v3.19.0 doesn't work with Raw(metadata={}) or Raw(type='binary/file'). Hence I tried Field() and it works fine.
To validate JPG, PNG files, I do that while #validates_schema by catching it's file type. I have also saved the file after in #post_load method.
class MySchema(Schema):
icon_url = Field(metadata={'type': 'string', 'format': 'byte'}, allow_none=True)
#validates_schema
def validate_uploaded_file(self, in_data, **kwargs):
errors = {}
file: FileStorage = in_data.get("icon_url", None)
if file is None:
# if any file is not uploaded, skip validation
pass
elif type(file) != FileStorage:
errors["icon_url"] = [
f"Invalid content. Only PNG, JPG/JPEG files accepted"]
raise ValidationError(errors)
elif file.content_type not in {"image/jpeg", "image/png"}:
errors["icon_url"] = [
f"Invalid file_type: {file.content_type}. Only PNG, JPG/JPEG images accepted."]
raise ValidationError(errors)
return in_data
#post_load
def post_load(self, loaded_obj, **kwargs):
if loaded_obj.icon_url:
sec_filename = secure_filename(
f'{loaded_obj.name}.{loaded_obj.icon_url.filename.split(".")[-1]}')
loaded_obj.icon_url.save(
f"{current_app.config['PUBLIC_IMAGES_FOLDER']}{sec_filename}")
loaded_obj.icon_url = f'{current_app.config["PUBLIC_IMAGES_URL"]}{sec_filename}'
return loaded_obj
I have a string file_id of my file that is stored in mongodb in a fs collection (GridFs).
I need to store the file as a mongoengine FileField in a Document, and then return the file to an endpoint ... so access the file's content, content_type etc.
I am not sure how to create a FileField instance with the GridFs string id? And is it possible to get the content and content_type from the FileField?
The tutorials that I have seen all involve creating the FileField by writing the contents to mongodb, the difference is my contents are already in the GridFs and I have the string id.
class Test(Document):
file = FileField()
test = Test()
test.upload.put(image_file, content_type='image/png')
So far I have been able to create a GridFsProxy object using the id, and can read the file using this.
class Test(Document):
file = FileField()
file_id = StringField() # bb2832e0-2ca4-44bc-8b1b-e01a77003b92
file_proxy = GridFSProxy(Test.file_id)
file_proxy.read() # Gives me the file content
file_proxy.get(file_id).content_type #can return name, length etc.
test = Test()
test.file = file_proxy.read() # in mongodb I see it as an ObjectID
If i store the read() results of the GridFSProxy into the FileField(); it is stored in MongoDb as an ObjectID and then later when I retrieve the object I don't seem to be able to get the content_type of the file.
I need the content_type as it is important for how I return the file contents.
I'm not sure how to create the FileField using just the file_id and then use it when i retrieve the Document.
Any insight into using the FileField (and GridFSProxy) will be helpful.
The FileField is basically just a reference (ObjectId) pointing to the actual grid fs documents (stored in fs.chunks/fs.files). Accessing the content_type should be straightforward, you shouldn't have to use the GridFSProxy class at all, see below:
from mongoengine import *
class Test(Document):
file = FileField()
test = Test()
image_bytes = open("path/to/image.png", "rb")
test.file.put(image_bytes, content_type='image/png', filename='test123.png')
test.save()
Test.objects.as_pymongo() # [{u'_id': ObjectId('5cdac41d992db9bfcaa870df'), u'file': ObjectId('5cdac419992db9bfcaa870dd')}]
t = Test.objects.first()
t.file # <GridFSProxy: 5cdac419992db9bfcaa870dd>
t.file.content_type # 'image/png'
t.file.filename # 'test123.png'
content = t.file.read()
I am trying to create a form where users will be allowed to upload any image file + SWF files. Django's ImageField does not support SWF so I need to override it.
What I want to do is to check if the file is SWF, if True, return it. If it's not SWF, call the original method which will take care of the file validation.
However, I am not sure how to implement that. Here is an example of what I am trying to achieve, but it does not work:
from hexagonit import swfheader
class SwfImageField(forms.ImageField):
def to_python(self, data):
try:
swfheader.parse(data)
return data
except:
return super(SwfImageField, self).to_python(data)
What is actually does is allowing only SWF files at the moment.
An alternative and possibly easiest solution is to use a standard FileField with a custom validator:
def my_validate(value):
ext = os.path.splitext(value.name)[1] # [0] returns path filename
valid = ['.jpg', '.swf']
if ext not in valid:
raise ValidationError("Unsupported file extension.")
class MyForm(forms.Form):
file = forms.FileField(validators=[my_validate])
may be it will be useful:
from django.core.validators import FileExtensionValidator
class MyModel(models.Model):
......
my_field = models.FileField('Name', validators=[FileExtensionValidator(['svg', 'jpg', 'png', '.....'])])