We Keep Coding

django opened file type to python opened file type

I have a django app that asks the user to upload an image.
I get the image from html django.
This image I pass to the python script as a parameter. I did a lot of stuff with this image (like using the PIL libraries), the class of the parameter is:
'django.core.files.uploadedfile.InMemoryUploadedFile'
But the problem comes when I try to use one function that ask for the predeterminate type of .open() of python, that is:
'_io.BufferedReader'
Concretely, the function I'm using is:
block_blob_service.create_blob_from_stream() (a Microsoft Azure function)
So my question is, can I convert from django opened file type to python opened file type? It may be without saving the file and opening again.
And, if by any chance, somebody has worked with this library, I've also tried block_blob_service.create_blob_from_bytes() and it's not working (to convert from django to bytes I've just done img = django_input.read() (I get a Bytes type) and block_blob_service.create_blob_from_path(), is not an option, because I can't get the path of the file, nor I don't want to save the image and get a new path.

Just according to the source code for django.core.files.uploadedfile, class InMemoryUploadedFile inherit from class UploadedFile which inherit from django.core.files.base.File, as the code and figure below said.
from django.core.files.base import File
class UploadedFile(File):
"""
An abstract uploaded file (``TemporaryUploadedFile`` and
``InMemoryUploadedFile`` are the built-in concrete subclasses).
An ``UploadedFile`` object behaves somewhat like a file object and
represents some file data that the user submitted with a form.
"""
def __init__(self, file=None, name=None, content_type=None, size=None, charset=None, content_type_extra=None):
super().__init__(file, name)
self.size = size
self.content_type = content_type
self.charset = charset
self.content_type_extra = content_type_extra
def __repr__(self):
return "<%s: %s (%s)>" % (self.__class__.__name__, self.name, self.content_type)
def _get_name(self):
return self._name
def _set_name(self, name):
# Sanitize the file name so that it can't be dangerous.
if name is not None:
# Just use the basename of the file -- anything else is dangerous.
name = os.path.basename(name)
# File names longer than 255 characters can cause problems on older OSes.
if len(name) > 255:
name, ext = os.path.splitext(name)
ext = ext[:255]
name = name[:255 - len(ext)] + ext
self._name = name
name = property(_get_name, _set_name)
class InMemoryUploadedFile(UploadedFile):
"""
A file uploaded into memory (i.e. stream-to-memory).
"""
def __init__(self, file, field_name, name, content_type, size, charset, content_type_extra=None):
super().__init__(file, name, content_type, size, charset, content_type_extra)
self.field_name = field_name
def open(self, mode=None):
self.file.seek(0)
return self
def chunks(self, chunk_size=None):
self.file.seek(0)
yield self.read()
def multiple_chunks(self, chunk_size=None):
# Since it's in memory, we'll never have multiple chunks.
return False
The figure below comes from https://docs.djangoproject.com/en/2.2/ref/files/uploads/
So you can get the data bytes from InMemoryUploadedFile object and pass it to the function block_blob_service.create_blob_from_bytes.
Meanwhile, as I known, it's not a good idea. The simple solution for creating a blob from the uploaded file in Django is to use django-storages with Azure Storage backend, please see its document about Azure Storage to know how to use. And there is the existing similar SO thread Django Azure upload file to blob storage which you can refer to.

Scrapy - How to have a custom path for multiple file URLs in one item when uploading to S3?

From my understanding is that in order for me to use Files Pipeline I need to include these to my settings.py:
AWS_ACCESS_KEY_ID = 'access key'
AWS_SECRET_ACCESS_KEY= 'secret'
FILES_STORE = 's3://bucket／'
And I need to add these to my Item object
file_urls = scrapy.Field()
files = scrapy.Field()
and the result would be something like this:
{'file_urls': ['http://i.stack.imgur.com/tKsDb.png',
'http://i.stack.imgur.com/NAGkl.png'],
'files': [{'checksum': 'b0974ea6c88740bed353206b279e0827',
'path': 'full/762f5682798c5854833316fa171c71166e284630.jpg',
'url': 'http://i.stack.imgur.com/tKsDb.png'},
{'checksum': '9a42f7bd1dc45840312fd49cd08e6a5c',
'path': 'full/615eabb7b61e79b96ea1ddb34a2ef55c8e0f7ec3.jpg',
'url': 'http://i.stack.imgur.com/NAGkl.png'}]}
From my understanding, the Files Pipeline will automatically populate the images field.
My question here is that is there anyway for me to change the 'path' value in the images field? In my case I want to store many different items under a different sub-directory each, and they are all scraped by the same spider, is there a way that I can do this, like creating and extending my own Files pipeline for example? If so, How would I approach this?
Also I need to mention that I pass 2 different URLs with 1 item and I would like them to be in different directories
(also yes I know they are images, I'm just using it as en example because they were the ones I found first)

Yes, this is possible, if you look at the source code for the scrapy files pipeline here
you will see that it has methods which you can override, one of them is the file_path method, which you override by adding it to your pipeline class like so
def file_path(self, request, response=None, info=None):
# start of deprecation warning block (can be removed in the future)
def _warn():
from scrapy.exceptions import ScrapyDeprecationWarning
import warnings
warnings.warn('FilesPipeline.file_key(url) method is deprecated,\
please use file_path(request, response=None, info=None) instead',
category=ScrapyDeprecationWarning, stacklevel=1)
# check if called from file_key with url as first argument
if not isinstance(request, Request):
_warn()
url = request
else:
url = request.url
# detect if file_key() method has been overridden
if not hasattr(self.file_key, '_base'):
_warn()
return self.file_key(url)
# end of deprecation warning block
# Modify the file path HERE to your own custom path
filename = request.meta['filename']
media_ext = 'jpg'
return '%s/%s/%s.%s' % \
(request.meta['image_category'],
request.meta['image_month'],
filename, media_ext)
The result of this would be a directory like:
vehicles/june/toyota.jpg
Where vehicles is the image_category, june is image_month and toyota is the filename
What this would do is if you look at the last few lines of the code [which is the only code I have added the rest is as the method is from the Scrapy source code]
# Modify the file path HERE to your own custom path
filename = request.meta['filename']
media_ext = 'jpg'
return '%s/%s/%s.%s' % \
(request.meta['image_category'],
request.meta['image_month'],
filename, media_ext)
is return a custom path
now this path depends on a few things, on the spider you can collect image meta fields like filename for image name, image category and anything else like date the image was take and use it in the pipeline to create a custom directory

How to dynamically select storage option for models.FileField?

Depending on the file extension, I want the file to be stored in a specific AWS bucket. I tried passing a function to the storage option, similar to how upload_to is dynamically defined.
However, this doesn't give the desired results. In my template, when I try href to document.docfile.url, the link doesn't work.
Checking in the shell, this happens
Document.objects.all()[0].docfile.storage.bucket
<Bucket: <function aws_bucket at 0x110672050>>
Document.objects.all()[0].docfile.storage.bucket_name
<function myproject.myapp.models.aws_bucket>
Desired behaviour would be
Document.objects.all()[0].docfile.storage.bucket_name
'asynch-uploader-txt'
Document.objects.all()[0].docfile.storage.bucket
<Bucket: asynch-uploader-txt>
This is my models.py file:
# -*- coding: utf-8 -*-
from django.db import models
from storages.backends.s3boto import S3BotoStorage
def upload_file_to(instance, filename):
import os
from django.utils.timezone import now
filename_base, filename_ext = os.path.splitext(filename)
return 'files/%s_%s%s' % (
filename_base,
now().strftime("%Y%m%d%H%M%S"),
filename_ext.lower(),
)
def aws_bucket(instance, filename):
import os
filename_base, filename_ext = os.path.splitext(filename)
return 'asynch-uploader-%s' %(filename_ext[1:])
class Document(models.Model):
docfile = models.FileField(upload_to=upload_file_to,storage=S3BotoStorage(bucket=aws_bucket))
Why is aws_bucket getting passed as a function and not a string, the way that upload_file_to is? How can I correct it?

For what you're trying to do you may be better off making a custom storage backend and just overriding the various bits of S3BotoStorage.
In particular if you make bucket_name a property you should be able to get the behavior you want.
EDIT:
To expand a bit on that, the source for S3BotoStorage.__init__ has the bucket as an optional argument. Additionally bucket when it's used in the class is a #param, making it easy to override. The following code is untested, but should be enough to give you a starting point
class MyS3BotoStorage(S3BotoStorage):
#property
def bucket(self):
if self._filename.endswith('.jpg'):
return self._get_or_create_bucket('SomeBucketName')
else:
return self._get_or_create_bucket('SomeSaneDefaultBucket')
def _save(self, name, content):
# This part might need some work to normalize the name and all...
self._filename = name
return super(MyS3BotoStorage, self)._save(name, content)

How to get file path of a add_static_view() in Pyramid

When I am adding a static view like this:
cfg = config.Configurator(...)
cfg.add_static_view(name='static', path='MyPgk:static')
# And I want to add a view for 'favicon.ico'.
cfg.add_route(name='favicon', pattern='/favicon.ico')
cfg.add_view(route_name='favicon', view='MyPgk.views.mymodule.favicon_view')
I am trying to add that favicon.ico annoying default path of /favicon.ico called by the browser if it's undefined in the webpage. I would like to use the example at http://docs.pylonsproject.org/projects/pyramid_cookbook/en/latest/files.html and modify it to have:
def favicon_view(request, cache=dict()):
if (not cache):
_path_to_MyPkg_static = __WHAT_GOES_HERE__
_icon = open(os.path.join(_path_to_MyPkg_static, 'favicon.ico')).read()
cache['response'] = Response(content_type='image/x-icon', body=_icon)
return cache['response']
Since, I can't really define the _here proposed in the example, how can I make it dependent to request to get the actual full path at runtime? Or do I really have to deal with:
_here = os.path.dirname(__file__)
_path_to_MyPkg_static = os.path.join(os.path.dirname(_here), 'static')
and having to be careful when I decide to refactor and put the view in another pkg or subpackage, or where-ever?
Something equivalent to request.static_path() but instead of getting the url path, to actually get a directory path:
request.static_file_path('static') -> /path/to/site-packages/MyPkg/static
Thanks,

You can use the pkg_resources module to make paths that are relative to Python modules (and thus, independent of the module that retrieves them). For example:
import pkg_resources
print pkg_resources.resource_filename('os.path', 'static/favicon.ico')
# 'C:\\Python27\\lib\\static\\favicon.ico'
Just substitute os.path with whatever module that is the parent of your static files.
EDIT: If you need to remember that 'static' route mapped to 'MyPkg:static', then the easiest way is to save it in some dictionary in the first place:
STATIC_ROUTES = {'static': 'MyPkg:static'}
for name, path in STATIC_ROUTES.iteritems():
cfg.add_static_view(name=name, path=path)
and then simply retrieve the path:
static_path = STATIC_ROUTES['static']
package, relative_path = static_path.split(':')
icon_path = pkg_resources.resource_filename(
package, os.path.join(relative_path, 'favicon.ico'))
If that's impossible, though (e.g. you don't have access to the cfg object), you can retrieve this path, it's just quite painful. Here's a sample function that uses undocumented calls (and so may change in future Pyramid versions) and ignores some additional settings (like route_prefix configuration variable):
def get_static_path(request, name):
from pyramid.config.views import StaticURLInfo
registrations = StaticURLInfo()._get_registrations(request.registry)
if not name.endswith('/'):
name = name + '/'
route_name = '__%s' % name
for _url, spec, reg_route_name in registrations:
print ':', reg_route_name
if reg_route_name == route_name:
return spec
In your case, it should work like this:
>>> get_static_path(request, 'static')
MyPkg:static/

Substitutions inside links in reST / Sphinx

I am using Sphinx to document a webservice that will be deployed in different servers. The documentation is full of URL examples for the user to click and they should just work. My problem is that the host, port and deployment root will vary and the documentation will have to be re-generated for every deployment.
I tried defining substitutions like this:
|base_url|/path
.. |base_url| replace:: http://localhost:8080
But the generated HTML is not what I want (doesn't include "/path" in the generated link):
http://localhost:8080/path
Does anybody know how to work around this?

New in Sphinx v1.0:
sphinx.ext.extlinks – Markup to shorten external links
https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html
The extension adds one new config value:
extlinks
This config value must be a dictionary of external sites, mapping unique short alias names to a base URL and a prefix. For example, to create an alias for the above mentioned issues, you would add
extlinks = {'issue':
('http://bitbucket.org/birkenfeld/sphinx/issue/%s', 'issue ')}
Now, you can use the alias name as a new role, e.g. :issue:`123`. This then inserts a link to http://bitbucket.org/birkenfeld/sphinx/issue/123. As you can see, the target given in the role is substituted in the base URL in the place of %s.
The link caption depends on the second item in the tuple, the prefix:
If the prefix is None, the link caption is the full URL.
If the prefix is the empty string, the link caption is the partial URL given in the role content (123 in this case.)
If the prefix is a non-empty string, the link caption is the partial URL, prepended by the prefix – in the above example, the link caption would be issue 123.
You can also use the usual “explicit title” syntax supported by other roles that generate links, i.e. :issue:`this issue <123>`. In this case, the prefix is not relevant.

I had a similar problem where I needed to substitute also URLs in image targets.
The extlinks do not expand when used as a value of image :target: attribute.
Eventually I wrote a custom sphinx transformation that rewrites URLs that start with a given prefix, in my case, http://mybase/. Here is a relevant code for conf.py:
from sphinx.transforms import SphinxTransform
class ReplaceMyBase(SphinxTransform):
default_priority = 750
prefix = 'http://mybase/'
def apply(self):
from docutils.nodes import reference, Text
baseref = lambda o: (
isinstance(o, reference) and
o.get('refuri', '').startswith(self.prefix))
basetext = lambda o: (
isinstance(o, Text) and o.startswith(self.prefix))
base = self.config.mybase.rstrip('/') + '/'
for node in self.document.traverse(baseref):
target = node['refuri'].replace(self.prefix, base, 1)
node.replace_attr('refuri', target)
for t in node.traverse(basetext):
t1 = Text(t.replace(self.prefix, base, 1), t.rawsource)
t.parent.replace(t, t1)
return
# end of class
def setup(app):
app.add_config_value('mybase', 'https://en.wikipedia.org/wiki', 'env')
app.add_transform(ReplaceMyBase)
return
This expands the following rst source to point to English wikipedia.
When conf.py sets mybase="https://es.wikipedia.org/wiki" the links would point to the Spanish wiki.
* inline link http://mybase/Helianthus
* `link with text <http://mybase/Helianthus>`_
* `link with separate definition`_
* image link |flowerimage|
.. _link with separate definition: http://mybase/Helianthus
.. |flowerimage| image:: https://upload.wikimedia.org/wikipedia/commons/f/f1/Tournesol.png
:target: http://mybase/Helianthus

Ok, here's how I did it. First, apilinks.py (the Sphinx extension):
from docutils import nodes, utils
def setup(app):
def api_link_role(role, rawtext, text, lineno, inliner, options={},
content=[]):
ref = app.config.apilinks_base + text
node = nodes.reference(rawtext, utils.unescape(ref), refuri=ref,
**options)
return [node], []
app.add_config_value('apilinks_base', 'http://localhost/', False)
app.add_role('apilink', api_link_role)
Now, in conf.py, add 'apilinks' to the extensions list and set an appropriate value for 'apilinks_base' (otherwise, it will default to 'http://localhost/'). My file looks like this:
extensions = ['sphinx.ext.autodoc', 'apilinks']
# lots of other stuff
apilinks_base = 'http://host:88/base/'
Usage:
:apilink:`path`
Output:
http://host:88/base/path

You can write a Sphinx extension that creates a role like
:apilink:`path`
and generates the link from that. I never did this, so I can't help more than giving this pointer, sorry. You should try to look at how the various roles are implemented. Many are very similar to what you need, I think.