Django image upload to Amazon S3 fails with boto - python

I have been struggling for a while trying to make media file uploads to Amazon S3 from Django work, and would be very thankful if anyone could help.
My app uploads image files without issues locally with a Sqlite3 database as well as Postgres, but when I try to upload my media files to S3, I get the following 500 error:
cannot concatenate 'str' and 'NoneType' objects
Update November 29th
It seems boto cannot find my bucket name, even if it is set in my settings. The 'NoneType' above comes from the following, which is from connection.py and bucket.py:
bucket: None
I have installed django-storages and boto, like described in the following articles:
Using Amazon S3 to Store your Django Site's Static and Media Files
Django-Storages and Amazon S3
Using AWS S3 to Store Static Assets and File Uploads
I tested uploading my media files to my bucket on S3 manually, and the database instances that I created locally links to correct media files from my bucket. From my point of view, the issue occurs when my django view tries to upload the image to S3 via boto (DEFAULT_FILE_STORAGE in my local_settings.py).
The following is in my local_settings.py:
INSTALLED_APPS = (
...
'storages',
)
# Local postgres database
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql_psycopg2",
"NAME": "name",
"USER": "username",
"PASSWORD": "pw",
"PORT": "5432",
}
}
STATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'
AWS_QUERYSTRING_AUTH = False
AWS_ACCESS_KEY_ID = 'my-key'
AWS_SECRET_ACCESS_KEY = 'my-key2'
AWS_STORAGE_BUCKET_NAME = 'my-bucket-name'
MEDIAFILES_LOCATION = 'media'
MEDIA_URL = 'http://%s.s3.amazonaws.com/media/' % AWS_STORAGE_BUCKET_NAME
DEFAULT_FILE_STORAGE = "storages.backends.s3boto.S3BotoStorage"
In my upload form in views.py I use something like:
photo = form.cleaned_data['photo']
...
recipe = Recipe(title=title, ..., photo=photo)
recipe.save()
I also tried adding a s3utils.py file like described in 2. Bucket policy and CORS is added like described in above articles. I granted "Everyone" access to upload/delete.
My requirements.txt contains:
Django==1.7.8
Pillow==2.6.1
awscli==1.7.36
boto==2.38.0
botocore==1.0.1
django-storages==1.1.8
whitenoise==1.0.6
...
I use Heroku for my app, and will need to store and host media files from S3.
Does anyone see what I am missing? Let me know if you need more details.
Thanks a lot in advance!
Traceback
Environment:
Request Method: POST
Request URL: http://127.0.0.1:5000/add/
Django Version: 1.7.8
Python Version: 2.7.10
Installed Applications:
('django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.admindocs',
'matboken_app',
'crispy_forms',
'PIL',
'django_summernote',
'djangoratings',
'imagekit',
'storages',
's3direct')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware')
Traceback:
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
111. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
21. return view_func(request, *args, **kwargs)
File "/Users/Kenneth/Dropbox/matboken/matboken_app/views.py" in addRecipe
678. recipe.save()
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/base.py" in save
589. force_update=force_update, update_fields=update_fields)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/base.py" in save_base
617. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/base.py" in _save_table
698. result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/base.py" in _do_insert
731. using=using, raw=raw)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/manager.py" in manager_method
92. return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/query.py" in _insert
921. return query.get_compiler(using=using).execute_sql(return_id)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
920. for sql, params in self.as_sql():
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in as_sql
878. for obj in self.query.objs
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/fields/files.py" in pre_save
301. file.save(file.name, file, save=False)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/fields/files.py" in save
89. self.name = self.storage.save(name, content)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/core/files/storage.py" in save
51. name = self._save(name, content)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/storages/backends/s3boto.py" in _save
385. key = self.bucket.get_key(encoded_name)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/boto/s3/bucket.py" in get_key
192. key, resp = self._get_key_internal(key_name, headers, query_args_l)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/boto/s3/bucket.py" in _get_key_internal
199. query_args=query_args)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/boto/s3/connection.py" in make_request
652. auth_path = self.calling_format.build_auth_path(bucket, key)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/boto/s3/connection.py" in build_auth_path
94. path = '/' + bucket
Exception Type: TypeError at /add/
Exception Value: cannot concatenate 'str' and 'NoneType' objects

Related

Django - SuspiciousFileOperation from image add on local development environment

I've been working on a personal photo website built on Django for the last several months and successfully deployed it with AWS. Til now I've done all my development on my Windows 10 desktop. I recently got a new Windows 10 laptop and have gotten stuck setting up a dev environment on it. I cloned my Github repo to my laptop and successfully gotten the site to run locally. However when I go to the admin page to upload a photo and every time I get a SuspiciousFileOperation exception. The joined path is located outside of the base path component. I've also gotten this error when running collectstatic. The settings are identical to my desktop setup, unless I'm overlooking an issue staring right at me.
I think it might have something to do with the way I set up the virtualenv or how my MEDIA variables are set up, though they are the same as what works on my desktop. I've read through the Django documentation but haven't found anything definitive. Or maybe someone could explain a bit more about setting up virtualenv? Does Django/Python need certain permissions I missed setting up on my laptop to move files around?
Below is the full error and the relevant code snippets.
Stack Trace:
Environment:
Request Method: POST
Request URL: http://127.0.0.1:8000/admin/PortfolioApp/image/add/
Django Version: 2.1.4
Python Version: 3.7.1
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'PortfolioApp.apps.PortfolioappConfig',
'storages']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback:
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\handlers\exception.py" in inner
34. response = get_response(request)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\handlers\base.py" in _get_response
126. response = self.process_exception_by_middleware(e, request)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\handlers\base.py" in _get_response
124. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\options.py" in wrapper
604. return self.admin_site.admin_view(view)(*args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\utils\decorators.py" in _wrapped_view
142. response = view_func(request, *args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\views\decorators\cache.py" in _wrapped_view_func
44. response = view_func(request, *args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\sites.py" in inner
223. return view(request, *args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\options.py" in add_view
1637. return self.changeform_view(request, None, form_url, extra_context)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\utils\decorators.py" in _wrapper
45. return bound_method(*args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\utils\decorators.py" in _wrapped_view
142. response = view_func(request, *args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\options.py" in changeform_view
1525. return self._changeform_view(request, object_id, form_url, extra_context)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\options.py" in _changeform_view
1564. self.save_model(request, new_object, form, not add)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\options.py" in save_model
1091. obj.save()
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\base.py" in save
718. force_update=force_update, update_fields=update_fields)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\base.py" in save_base
748. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\base.py" in _save_table
831. result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\base.py" in _do_insert
869. using=using, raw=raw)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\manager.py" in manager_method
82. return getattr(self.get_queryset(), name)(*args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\query.py" in _insert
1136. return query.get_compiler(using=using).execute_sql(return_id)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\sql\compiler.py" in execute_sql
1288. for sql, params in self.as_sql():
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\sql\compiler.py" in as_sql
1241. for obj in self.query.objs
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\sql\compiler.py" in <listcomp>
1241. for obj in self.query.objs
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\sql\compiler.py" in <listcomp>
1240. [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\sql\compiler.py" in pre_save_val
1192. return field.pre_save(obj, add=True)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\fields\files.py" in pre_save
288. file.save(file.name, file.file, save=False)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\fields\files.py" in save
87. self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\files\storage.py" in save
48. name = self.get_available_name(name, max_length=max_length)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\files\storage.py" in get_available_name
72. while self.exists(name) or (max_length and len(name) > max_length):
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\files\storage.py" in exists
308. return os.path.exists(self.path(name))
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\files\storage.py" in path
321. return safe_join(self.location, name)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\utils\_os.py" in safe_join
49. 'component ({})'.format(final_path, base_path))
Exception Type: SuspiciousFileOperation at /admin/PortfolioApp/image/add/
Exception Value: The joined path (C:\Users\Bryan\Documents\GitHub\Portfolio\media\pictures\VeselkaNYC-BJM.jpg) is located outside of the base path component (C:\Users\Bryan\Documents\GitHub\Portfolio\media\)
models.py
class Image(models.Model):
...
picture = models.ImageField(upload_to='pictures/', height_field='height', width_field='width', null=True)
...
settings.py
...
STATIC_URL = '/static/'
STATIC_ROOT = 'static/'
MEDIA_URL = '/media/'
MEDIA_ROOT = 'media/'
...
urls.py
...
if settings.DEBUG:
# Use static() to add url mapping to serve static files during development (only)
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
I've included what I think are the relevant code snippets but please ask if you need more to help me out. Any help is much appreciated.
EDIT:
Thanks to Giancarlo's suggestions, redefined my STATIC_ROOT and MEDIA_ROOT as follows:
STATIC_ROOT = os.join.path(BASE_DIR, "static")
MEDIA_ROOT = os.join.path(BASE_DIR, "media")
I think the settings on my desktop work because of a .pyc cache file containing the correct settings that were saved during the development process. That's the best I can come up with.
Change your STATIC_ROOT and MEDIA_ROOT to
STATIC_ROOT = os.path.join(BASE_DIR, "static/")
MEDIA_ROOT = os.path.join(BASE_DIR, "media/")
Because maybe Windows is searching at C:\media or C:\static
Edit: I think BASE_DIR is pre-defined in settings.py, but here is if it is missing
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
And to be sure, add a trailing slash to the upload_to
upload_to='pictures/'

django template errno 13

I have a problem with django template. I made new template file in pycharm and in code i think all is good. But i have errno 13 permission denied. Other templates are working fine.
Environment:
Request Method: GET
Request URL: http://127.0.0.1:8000/Doggo/2/results/
Django Version: 1.11
Python Version: 3.6.1
Installed Applications:
['Doggo.apps.DoggoConfig',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback:
File "C:\Python36-32\lib\site-packages\django\core\handlers\exception.py" in inner
41. response = get_response(request)
File "C:\Python36-32\lib\site-packages\django\core\handlers\base.py" in _get_response
217. response = self.process_exception_by_middleware(e, request)
File "C:\Python36-32\lib\site-packages\django\core\handlers\base.py" in _get_response
215. response = response.render()
File "C:\Python36-32\lib\site-packages\django\template\response.py" in render
107. self.content = self.rendered_content
File "C:\Python36-32\lib\site-packages\django\template\response.py" in rendered_content
82. template = self.resolve_template(self.template_name)
File "C:\Python36-32\lib\site-packages\django\template\response.py" in resolve_template
64. return select_template(template, using=self.using)
File "C:\Python36-32\lib\site-packages\django\template\loader.py" in select_template
48. return engine.get_template(template_name)
File "C:\Python36-32\lib\site-packages\django\template\backends\django.py" in get_template
39. return Template(self.engine.get_template(template_name), self)
File "C:\Python36-32\lib\site-packages\django\template\engine.py" in get_template
162. template, origin = self.find_template(template_name)
File "C:\Python36-32\lib\site-packages\django\template\engine.py" in find_template
136. name, template_dirs=dirs, skip=skip,
File "C:\Python36-32\lib\site-packages\django\template\loaders\base.py" in get_template
38. contents = self.get_contents(origin)
File "C:\Python36-32\lib\site-packages\django\template\loaders\filesystem.py" in get_contents
28. with io.open(origin.name, encoding=self.engine.file_charset) as fp:
Exception Type: PermissionError at /Doggo/2/results/
Exception Value: [Errno 13] Permission denied: 'C:\\Users\\noob1_000\\PycharmProjects\\eWybory\\Doggo\\templates'
i think i have permission to this folder because other templates are working.
I'm working on windows 10
I would recommend to go through this-
Also , the view that you uses the template are also in the same directory as other views since you said that other views are able to access their templates ?
Let me know , if this solves your problem
Maybe you are just forgetting to indicate the template path (template / app/index.html) in the views.py.

Django / xhtml2pdf - object has no attribute 'encode'

I'm having a hard time producing pdf's in Django. As per my previous question I'm running into the same error.
When I run the following code I get 'list' object has no attribute 'encode', a pdf is saved in the media folder but it is a plain text file.
I think the object that is being referred to is the sections queryset.
#login_required
def generate_pdf(request, slug):
# Prepare context
document = get_object_or_404(Document, slug=slug)
sections = \
get_list_or_404(Section.objects.filter
(associated_document__startswith=document.slug))
data = {'document': document, 'sections': sections}
# Render html content through html template with context
template = get_template('lld/lld_pdf.html')
html = template.render(Context(data))
file = open('/home/project/media/test.pdf', "w+b")
print type(document)
print type(sections)
print type(data)
print type(template)
print type(html)
print type(file)
pisaStatus = pisa.CreatePDF(html.encode('utf-8'), dest=file)
# Return PDF document through a Django HTTP response
file.seek(0)
pdf = file.read()
file.close() # Don't forget to close the file handle
return HttpResponse(pdf, mimetype='application/pdf')
cli output:
<class 'lld.models.Document'>
<type 'list'>
<type 'dict'>
<class 'django.template.backends.django.Template'>
<class 'django.utils.safestring.SafeText'>
<type 'file'>
and the error:
Environment:
Request Method: GET
Request URL: http://localhost:8001/lld/tesco-greenfield-datacenter-deployment/pdf/
Django Version: 1.8.2
Python Version: 2.7.6
Installed Applications:
('django_admin_bootstrapped',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'lld',
'registration')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.middleware.security.SecurityMiddleware')
Traceback:
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response
132. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/decorators.py" in _wrapped_view
22. return view_func(request, *args, **kwargs)
File "/home/vagrant/shared/RepeatableDesign/lld/views.py" in generate_pdf
162. pisaStatus = pisa.CreatePDF(html.encode('utf-8'), dest=file)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/document.py" in pisaDocument
89. encoding, context=context, xml_output=xml_output)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/document.py" in pisaStory
57. pisaParser(src, context, default_css, xhtml, encoding, xml_output)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/parser.py" in pisaParser
685. context.parseCSS()
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/context.py" in parseCSS
498. self.css = self.cssParser.parse(self.cssText)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/w3c/cssParser.py" in parse
434. src, stylesheet = self._parseStylesheet(src)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/w3c/cssParser.py" in _parseStylesheet
522. src, stylesheetImports = self._parseAtImports(src)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/w3c/cssParser.py" in _parseAtImports
606. stylesheet = self.cssBuilder.atImport(import_, mediums, self)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/w3c/css.py" in atImport
874. return cssParser.parseExternal(import_)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/context.py" in parseExternal
380. result = self.parse(cssFile.getData())
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/w3c/cssParser.py" in parse
434. src, stylesheet = self._parseStylesheet(src)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/w3c/cssParser.py" in _parseStylesheet
533. src, atResults = self._parseAtKeyword(src)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/w3c/cssParser.py" in _parseAtKeyword
655. src, result = self._parseAtFontFace(src)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/w3c/cssParser.py" in _parseAtFontFace
785. result = [self.cssBuilder.atFontFace(properties)]
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/context.py" in atFontFace
173. src = self.c.getFile(data["src"], relative=self.c.cssParser.rootPath)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/context.py" in getFile
823. return getFile(name, relative or self.pathDirectory)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/util.py" in getFile
635. file = pisaFileObject(*a, **kw)
File "/usr/local/lib/python2.7/dist-packages/xhtml2pdf/util.py" in __init__
516. uri = uri.encode('utf-8')
Exception Type: AttributeError at /lld/tesco-greenfield-datacenter-deployment/pdf/
Exception Value: 'list' object has no attribute 'encode'
I can't even believe I've spent a few days crying and tearing my hair out because I was using a Google font. Yep, once I removed <link href="http://fonts.googleapis.com/css?family=Lato:400,700" rel="stylesheet" type="text/css"> from my template, the error vamoosed!

Error in import FloatField, using django-import-export

I am using django-import-export for import csv file. I have a FloatField in my model :
models.py
purchase_price = models.FloatField(null=True, blank=True)
When I import csv file with blank value, it throws an error :
ValueError at /admin/csv_imp/book/process_import/
could not convert string to float:
Request Method: POST
Request URL: http://localhost:8000/admin/csv_imp/book/process_import/
Django Version: 1.7.1
Exception Type: ValueError
Exception Value:
could not convert string to float:
Exception Location: /home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py in get_prep_value, line 1550
Python Executable: /home/bgdev/Desktop/virtualenvs/django17/bin/python
Python Version: 2.7.6
Python Path:
['/home/bgdev/Desktop/virtualenvs/django17/test_pro',
'/home/bgdev/Desktop/virtualenvs/django17/src/admin-bootstrap',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7/plat-i386-linux-gnu',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7/lib-tk',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7/lib-old',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7/lib-dynload',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-i386-linux-gnu',
'/usr/lib/python2.7/lib-tk',
'/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7/site-packages']
Server time: Tue, 9 Dec 2014 05:33:36 +0000
Environment:
Request Method: POST
Request URL: http://localhost:8000/admin/csv_imp/book/process_import/
Django Version: 1.7.1
Python Version: 2.7.6
Installed Applications:
('bootstrap_admin',
'import_export',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'csv_imp')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware')
Traceback:
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
111. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view
105. response = view_func(request, *args, **kwargs)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
52. response = view_func(request, *args, **kwargs)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/contrib/admin/sites.py" in inner
204. return view(request, *args, **kwargs)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/import_export/admin.py" in process_import
130. raise_errors=True)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/import_export/resources.py" in import_data
359. six.reraise(*sys.exc_info())
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/import_export/resources.py" in import_data
345. self.save_instance(instance, real_dry_run)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/import_export/resources.py" in save_instance
163. instance.save()
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/base.py" in save
591. force_update=force_update, update_fields=update_fields)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/base.py" in save_base
619. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/base.py" in _save_table
681. forced_update)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/base.py" in _do_update
725. return filtered._update(values) > 0
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/query.py" in _update
600. return query.get_compiler(self.db).execute_sql(CURSOR)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
1004. cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
775. sql, params = self.as_sql()
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in as_sql
969. val = field.get_db_prep_save(val, connection=self.connection)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py" in get_db_prep_save
627. prepared=False)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py" in get_db_prep_value
619. value = self.get_prep_value(value)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py" in get_prep_value
1550. return float(value)
Exception Type: ValueError at /admin/csv_imp/book/process_import/
Exception Value: could not convert string to float:
I found similar issue with and IntegerField. Being the last column in the csv file, it turns out that I was introducing an extra End of Line character in the dataset, even if the field had an empty value. Removing the EOL character from every line before adding it to the dataset fixed the issue:
for line in file.readlines():
line = line.rstrip('\r\n')
dataset.append(tuple(item for item in line.split(',')))
This link will show you what null and blank do: differentiate null=True, blank=True in django
If you can arrange your csv file to have all of the null values at the end, you can just remove the commas. That might work. Or perhaps create a temporary model that assigns those fields to a Charfield. Then copy appropriate fields to your final model
This is untested:
update <table> set <field> = 0 where id in (select id from <table> where <field> = "");
Fill in your table name and the field name as appropriate. You can run this from your sql command line program

Uploading to Google Cloud Storage from Django on App Engine

I'm using Django 1.5 on Google App Engine and I'm trying to upload files to Google Cloud Storage. I'm using the gcs the library and have written a custom file upload handler that I've registered in settings.py as my only file uploader. I can see my file being correctly uploaded in the blobstore viewer in my development environment but once form.save() is called in views.py I get an exception thrown saying that it's a read only file system? I know Google App Engine doesn't allow access to the file system which is why I'm using GCS in the first place!
Is there something I need to do to stop Django from trying to save the file to disk?
The relevant code is attached in this gist.
Thanks :)
Stack Trace:
Environment:
Request Method: POST
Request URL: http://localhost:8080/cms/media/add
Django Version: 1.5
Python Version: 2.7.5
Installed Applications:
('django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.staticfiles',
'django.contrib.messages',
'api',
'cms',
'frontend')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware')
Traceback:
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/core/handlers/base.py" in get_response
115. response = callback(request, *callback_args, **callback_kwargs)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/contrib/auth/decorators.py" in _wrapped_view
25. return view_func(request, *args, **kwargs)
File "/Users/james/Dropbox/University/Year 4/Advanced Development/assignment/cms/views.py" in media_add_or_edit
44. form.save()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/forms/models.py" in save
370. fail_message, commit, construct=False)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/forms/models.py" in save_instance
87. instance.save()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/db/models/base.py" in save
546. force_update=force_update, update_fields=update_fields)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/db/models/base.py" in save_base
650. result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/db/models/manager.py" in _insert
215. return insert_query(self.model, objs, fields, **kwargs)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/db/models/query.py" in insert_query
1673. return query.get_compiler(using=using).execute_sql(return_id)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/db/models/sql/compiler.py" in execute_sql
936. for sql, params in self.as_sql():
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/db/models/sql/compiler.py" in as_sql
894. for obj in self.query.objs
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/db/models/fields/files.py" in pre_save
250. file.save(file.name, file, save=False)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/db/models/fields/files.py" in save
86. self.name = self.storage.save(name, content)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/core/files/storage.py" in save
48. name = self._save(name, content)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django-1.5/django/core/files/storage.py" in _save
198. fd = os.open(full_path, flags, 0o666)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/devappserver2/python/stubs.py" in fake_open
71. raise OSError(errno.EROFS, 'Read-only file system', filename)
Exception Type: OSError at /cms/media/add
Exception Value: [Errno 30] Read-only file system: u'/Users/james/Dropbox/University/Year 4/Advanced Development/assignment/IMG_0746.jpg'
I eventually solved this by moving the FileField from my Model into my ModelForm like so:
# Used for uploading media that forms part of a story
class Media(models.Model):
title = models.CharField(max_length=100)
type = models.CharField(max_length=5, choices=MEDIA_TYPES)
content = models.TextField()
date_created = models.DateTimeField(auto_now_add=True)
# Used to convert the media model to a form in the cms
class MediaForm(forms.ModelForm):
file = forms.FileField()
class Meta:
model = Media
# Don't show the date created field because we want that to be set automatically
exclude = ('date_created', 'content',)
I'm sure I tried that before but it appears to have fixed my issue, hopefully this might help someone else who runs into the same problem.

Categories