We Keep Coding

Django - SuspiciousFileOperation from image add on local development environment

I've been working on a personal photo website built on Django for the last several months and successfully deployed it with AWS. Til now I've done all my development on my Windows 10 desktop. I recently got a new Windows 10 laptop and have gotten stuck setting up a dev environment on it. I cloned my Github repo to my laptop and successfully gotten the site to run locally. However when I go to the admin page to upload a photo and every time I get a SuspiciousFileOperation exception. The joined path is located outside of the base path component. I've also gotten this error when running collectstatic. The settings are identical to my desktop setup, unless I'm overlooking an issue staring right at me.
I think it might have something to do with the way I set up the virtualenv or how my MEDIA variables are set up, though they are the same as what works on my desktop. I've read through the Django documentation but haven't found anything definitive. Or maybe someone could explain a bit more about setting up virtualenv? Does Django/Python need certain permissions I missed setting up on my laptop to move files around?
Below is the full error and the relevant code snippets.
Stack Trace:
Environment:
Request Method: POST
Request URL: http://127.0.0.1:8000/admin/PortfolioApp/image/add/
Django Version: 2.1.4
Python Version: 3.7.1
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'PortfolioApp.apps.PortfolioappConfig',
'storages']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware']
Traceback:
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\handlers\exception.py" in inner
34. response = get_response(request)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\handlers\base.py" in _get_response
126. response = self.process_exception_by_middleware(e, request)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\handlers\base.py" in _get_response
124. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\options.py" in wrapper
604. return self.admin_site.admin_view(view)(*args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\utils\decorators.py" in _wrapped_view
142. response = view_func(request, *args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\views\decorators\cache.py" in _wrapped_view_func
44. response = view_func(request, *args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\sites.py" in inner
223. return view(request, *args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\options.py" in add_view
1637. return self.changeform_view(request, None, form_url, extra_context)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\utils\decorators.py" in _wrapper
45. return bound_method(*args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\utils\decorators.py" in _wrapped_view
142. response = view_func(request, *args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\options.py" in changeform_view
1525. return self._changeform_view(request, object_id, form_url, extra_context)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\options.py" in _changeform_view
1564. self.save_model(request, new_object, form, not add)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\contrib\admin\options.py" in save_model
1091. obj.save()
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\base.py" in save
718. force_update=force_update, update_fields=update_fields)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\base.py" in save_base
748. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\base.py" in _save_table
831. result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\base.py" in _do_insert
869. using=using, raw=raw)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\manager.py" in manager_method
82. return getattr(self.get_queryset(), name)(*args, **kwargs)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\query.py" in _insert
1136. return query.get_compiler(using=using).execute_sql(return_id)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\sql\compiler.py" in execute_sql
1288. for sql, params in self.as_sql():
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\sql\compiler.py" in as_sql
1241. for obj in self.query.objs
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\sql\compiler.py" in <listcomp>
1241. for obj in self.query.objs
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\sql\compiler.py" in <listcomp>
1240. [self.prepare_value(field, self.pre_save_val(field, obj)) for field in fields]
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\sql\compiler.py" in pre_save_val
1192. return field.pre_save(obj, add=True)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\fields\files.py" in pre_save
288. file.save(file.name, file.file, save=False)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\db\models\fields\files.py" in save
87. self.name = self.storage.save(name, content, max_length=self.field.max_length)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\files\storage.py" in save
48. name = self.get_available_name(name, max_length=max_length)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\files\storage.py" in get_available_name
72. while self.exists(name) or (max_length and len(name) > max_length):
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\files\storage.py" in exists
308. return os.path.exists(self.path(name))
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\core\files\storage.py" in path
321. return safe_join(self.location, name)
File "C:\Users\Bryan\Envs\dj\lib\site-packages\django\utils\_os.py" in safe_join
49. 'component ({})'.format(final_path, base_path))
Exception Type: SuspiciousFileOperation at /admin/PortfolioApp/image/add/
Exception Value: The joined path (C:\Users\Bryan\Documents\GitHub\Portfolio\media\pictures\VeselkaNYC-BJM.jpg) is located outside of the base path component (C:\Users\Bryan\Documents\GitHub\Portfolio\media\)
models.py
class Image(models.Model):
...
picture = models.ImageField(upload_to='pictures/', height_field='height', width_field='width', null=True)
...
settings.py
...
STATIC_URL = '/static/'
STATIC_ROOT = 'static/'
MEDIA_URL = '/media/'
MEDIA_ROOT = 'media/'
...
urls.py
...
if settings.DEBUG:
# Use static() to add url mapping to serve static files during development (only)
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
I've included what I think are the relevant code snippets but please ask if you need more to help me out. Any help is much appreciated.
EDIT:
Thanks to Giancarlo's suggestions, redefined my STATIC_ROOT and MEDIA_ROOT as follows:
STATIC_ROOT = os.join.path(BASE_DIR, "static")
MEDIA_ROOT = os.join.path(BASE_DIR, "media")
I think the settings on my desktop work because of a .pyc cache file containing the correct settings that were saved during the development process. That's the best I can come up with.

Change your STATIC_ROOT and MEDIA_ROOT to
STATIC_ROOT = os.path.join(BASE_DIR, "static/")
MEDIA_ROOT = os.path.join(BASE_DIR, "media/")
Because maybe Windows is searching at C:\media or C:\static
Edit: I think BASE_DIR is pre-defined in settings.py, but here is if it is missing
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
And to be sure, add a trailing slash to the upload_to
upload_to='pictures/'

Django -- Models don't exist, but Django still loads them

Im getting a ProgrammingError when I try to delete a User object, this is wherever User.delete occurs, it even happens in the admin.
The Error
Django apparently 'thinks' that there's a relationship between auth_user and apiHandlers_cardholders which doesn't exist anywhere, the database doesn't have a column for this, and there is no models.[y nor admin.py in the app named apiHandlers , but Django expects it to, there is even a section of the Admin site devoted to it. This only started happening when I migrated from SQLite to PostGreSQL.
At the moment all the local dependencies (models, admin, forms and fields) are in an app called helpers, I'm not sure if this is efficient, but it keeps everything in one place and avoids complications.
UPDATE 0.1
No. One of the probably gonna be many updates
I used to have a problem when migrating, related to Django Rest Framework, but I eventually managed to fix it by a quick pip install to a new update of restframework, it's no longer giving me any error while making migrations and migrating.
Below is the current traceback I'm getting from the Django admin, it's basically the same in the other deletion methods.
`
Environment:
Request Method: GET
Request URL: http://localhost:8000/admin/auth/user/3/delete/
Django Version: 1.9.2
Python Version: 2.7.10
Installed Applications:
['django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'rest_framework',
'corsheaders',
'oauth2_provider',
'apiHandlers',
'frontEnd',
'helpers',
'nonUser']
Installed Middleware:
['django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'corsheaders.middleware.CorsMiddleware',
'oauth2_provider.middleware.OAuth2TokenMiddleware']
Traceback:
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
149. response = self.process_exception_by_middleware(e, request)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
147. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/contrib/admin/options.py" in wrapper
541. return self.admin_site.admin_view(view)(*args, **kwargs)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view
149. response = view_func(request, *args, **kwargs)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
57. response = view_func(request, *args, **kwargs)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/contrib/admin/sites.py" in inner
244. return view(request, *args, **kwargs)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapper
67. return bound_func(*args, **kwargs)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view
149. response = view_func(request, *args, **kwargs)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/utils/decorators.py" in bound_func
63. return func.__get__(self, type(self))(*args2, **kwargs2)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/utils/decorators.py" in inner
184. return func(*args, **kwargs)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/contrib/admin/options.py" in delete_view
1629. [obj], opts, request.user, self.admin_site, using)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/contrib/admin/utils.py" in get_deleted_objects
128. collector.collect(objs)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/contrib/admin/utils.py" in collect
193. return super(NestedObjects, self).collect(objs, source_attr=source_attr, **kwargs)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/db/models/deletion.py" in collect
230. elif sub_objs:
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/db/models/query.py" in __nonzero__
266. return type(self).__bool__(self)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/db/models/query.py" in __bool__
262. self._fetch_all()
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/db/models/query.py" in _fetch_all
1074. self._result_cache = list(self.iterator())
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/db/models/query.py" in __iter__
52. results = compiler.execute_sql()
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
848. cursor.execute(sql, params)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/db/backends/utils.py" in execute
79. return super(CursorDebugWrapper, self).execute(sql, params)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/db/backends/utils.py" in execute
64. return self.cursor.execute(sql, params)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/db/utils.py" in __exit__
95. six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/Users/Mason/Documents/dev/FlowFrameWork/flow-framework/env/lib/python2.7/site-packages/django/db/backends/utils.py" in execute
64. return self.cursor.execute(sql, params)
Exception Type: ProgrammingError at /admin/auth/user/3/delete/
Exception Value: relation "apiHandlers_cardholder" does not exist
LINE 1: ...user"."is_active", "auth_user"."date_joined" FROM "apiHandle...
^
`
UPDATE 0.2
I am using ./manage.py runserver (Mac) to run it, so far I haven't managed to test it with Nginx and WSGI on my server.
I am not getting a warning saying that it could affect another model, in the admin, for example, I am just getting this error immediately.

This is very likely to be because you have a stray models.pyc file somewhere. Clear out all the *.pyc files. You can try something like this if you are on linux:
find . -name '*pyc' -exec 'rm' '{}' ';'
Follow that by ./manage.py makemigrations apihandlers and then ./manage.py migrate

Django image upload to Amazon S3 fails with boto

I have been struggling for a while trying to make media file uploads to Amazon S3 from Django work, and would be very thankful if anyone could help.
My app uploads image files without issues locally with a Sqlite3 database as well as Postgres, but when I try to upload my media files to S3, I get the following 500 error:
cannot concatenate 'str' and 'NoneType' objects
Update November 29th
It seems boto cannot find my bucket name, even if it is set in my settings. The 'NoneType' above comes from the following, which is from connection.py and bucket.py:
bucket: None
I have installed django-storages and boto, like described in the following articles:
Using Amazon S3 to Store your Django Site's Static and Media Files
Django-Storages and Amazon S3
Using AWS S3 to Store Static Assets and File Uploads
I tested uploading my media files to my bucket on S3 manually, and the database instances that I created locally links to correct media files from my bucket. From my point of view, the issue occurs when my django view tries to upload the image to S3 via boto (DEFAULT_FILE_STORAGE in my local_settings.py).
The following is in my local_settings.py:
INSTALLED_APPS = (
...
'storages',
)
# Local postgres database
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql_psycopg2",
"NAME": "name",
"USER": "username",
"PASSWORD": "pw",
"PORT": "5432",
}
}
STATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'
AWS_QUERYSTRING_AUTH = False
AWS_ACCESS_KEY_ID = 'my-key'
AWS_SECRET_ACCESS_KEY = 'my-key2'
AWS_STORAGE_BUCKET_NAME = 'my-bucket-name'
MEDIAFILES_LOCATION = 'media'
MEDIA_URL = 'http://%s.s3.amazonaws.com/media/' % AWS_STORAGE_BUCKET_NAME
DEFAULT_FILE_STORAGE = "storages.backends.s3boto.S3BotoStorage"
In my upload form in views.py I use something like:
photo = form.cleaned_data['photo']
...
recipe = Recipe(title=title, ..., photo=photo)
recipe.save()
I also tried adding a s3utils.py file like described in 2. Bucket policy and CORS is added like described in above articles. I granted "Everyone" access to upload/delete.
My requirements.txt contains:
Django==1.7.8
Pillow==2.6.1
awscli==1.7.36
boto==2.38.0
botocore==1.0.1
django-storages==1.1.8
whitenoise==1.0.6
...
I use Heroku for my app, and will need to store and host media files from S3.
Does anyone see what I am missing? Let me know if you need more details.
Thanks a lot in advance!
Traceback
Environment:
Request Method: POST
Request URL: http://127.0.0.1:5000/add/
Django Version: 1.7.8
Python Version: 2.7.10
Installed Applications:
('django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.admindocs',
'matboken_app',
'crispy_forms',
'PIL',
'django_summernote',
'djangoratings',
'imagekit',
'storages',
's3direct')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware')
Traceback:
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
111. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
21. return view_func(request, *args, **kwargs)
File "/Users/Kenneth/Dropbox/matboken/matboken_app/views.py" in addRecipe
678. recipe.save()
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/base.py" in save
589. force_update=force_update, update_fields=update_fields)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/base.py" in save_base
617. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/base.py" in _save_table
698. result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/base.py" in _do_insert
731. using=using, raw=raw)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/manager.py" in manager_method
92. return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/query.py" in _insert
921. return query.get_compiler(using=using).execute_sql(return_id)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
920. for sql, params in self.as_sql():
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in as_sql
878. for obj in self.query.objs
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/fields/files.py" in pre_save
301. file.save(file.name, file, save=False)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/db/models/fields/files.py" in save
89. self.name = self.storage.save(name, content)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/django/core/files/storage.py" in save
51. name = self._save(name, content)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/storages/backends/s3boto.py" in _save
385. key = self.bucket.get_key(encoded_name)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/boto/s3/bucket.py" in get_key
192. key, resp = self._get_key_internal(key_name, headers, query_args_l)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/boto/s3/bucket.py" in _get_key_internal
199. query_args=query_args)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/boto/s3/connection.py" in make_request
652. auth_path = self.calling_format.build_auth_path(bucket, key)
File "/Users/Kenneth/Dropbox/matboken/env_1.8/lib/python2.7/site-packages/boto/s3/connection.py" in build_auth_path
94. path = '/' + bucket
Exception Type: TypeError at /add/
Exception Value: cannot concatenate 'str' and 'NoneType' objects

Error in import FloatField, using django-import-export

I am using django-import-export for import csv file. I have a FloatField in my model :
models.py
purchase_price = models.FloatField(null=True, blank=True)
When I import csv file with blank value, it throws an error :
ValueError at /admin/csv_imp/book/process_import/
could not convert string to float:
Request Method: POST
Request URL: http://localhost:8000/admin/csv_imp/book/process_import/
Django Version: 1.7.1
Exception Type: ValueError
Exception Value:
could not convert string to float:
Exception Location: /home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py in get_prep_value, line 1550
Python Executable: /home/bgdev/Desktop/virtualenvs/django17/bin/python
Python Version: 2.7.6
Python Path:
['/home/bgdev/Desktop/virtualenvs/django17/test_pro',
'/home/bgdev/Desktop/virtualenvs/django17/src/admin-bootstrap',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7/plat-i386-linux-gnu',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7/lib-tk',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7/lib-old',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7/lib-dynload',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-i386-linux-gnu',
'/usr/lib/python2.7/lib-tk',
'/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages',
'/home/bgdev/Desktop/virtualenvs/django17/lib/python2.7/site-packages']
Server time: Tue, 9 Dec 2014 05:33:36 +0000
Environment:
Request Method: POST
Request URL: http://localhost:8000/admin/csv_imp/book/process_import/
Django Version: 1.7.1
Python Version: 2.7.6
Installed Applications:
('bootstrap_admin',
'import_export',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'csv_imp')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware')
Traceback:
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
111. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view
105. response = view_func(request, *args, **kwargs)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
52. response = view_func(request, *args, **kwargs)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/contrib/admin/sites.py" in inner
204. return view(request, *args, **kwargs)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/import_export/admin.py" in process_import
130. raise_errors=True)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/import_export/resources.py" in import_data
359. six.reraise(*sys.exc_info())
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/import_export/resources.py" in import_data
345. self.save_instance(instance, real_dry_run)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/import_export/resources.py" in save_instance
163. instance.save()
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/base.py" in save
591. force_update=force_update, update_fields=update_fields)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/base.py" in save_base
619. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/base.py" in _save_table
681. forced_update)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/base.py" in _do_update
725. return filtered._update(values) > 0
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/query.py" in _update
600. return query.get_compiler(self.db).execute_sql(CURSOR)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
1004. cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
775. sql, params = self.as_sql()
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in as_sql
969. val = field.get_db_prep_save(val, connection=self.connection)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py" in get_db_prep_save
627. prepared=False)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py" in get_db_prep_value
619. value = self.get_prep_value(value)
File "/home/bgdev/Desktop/virtualenvs/django17/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py" in get_prep_value
1550. return float(value)
Exception Type: ValueError at /admin/csv_imp/book/process_import/
Exception Value: could not convert string to float:

I found similar issue with and IntegerField. Being the last column in the csv file, it turns out that I was introducing an extra End of Line character in the dataset, even if the field had an empty value. Removing the EOL character from every line before adding it to the dataset fixed the issue:
for line in file.readlines():
line = line.rstrip('\r\n')
dataset.append(tuple(item for item in line.split(',')))

This link will show you what null and blank do: differentiate null=True, blank=True in django
If you can arrange your csv file to have all of the null values at the end, you can just remove the commas. That might work. Or perhaps create a temporary model that assigns those fields to a Charfield. Then copy appropriate fields to your final model
This is untested:
update <table> set <field> = 0 where id in (select id from <table> where <field> = "");
Fill in your table name and the field name as appropriate. You can run this from your sql command line program

django when adding data from admin interface: TypeError at admin/... unicode object not callable

I'm getting an error when trying to save to the following model using the admin interface:
models.py
class Answer(models.Model):
a = models.TextField(primary_key=True)
gloss = models.TextField(blank=True)
clean = models.TextField(blank=True)
count = models.IntegerField(blank=True)
p = models.IntegerField(blank=True)
def __unicode__(self):
return u"%s" % self.a
class Meta:
db_table = u'answers'
here's the error message that shows up on the admin interface:
Environment:
Request Method: POST
Request URL: http://localhost:8000/admin/emotions/answer/add/
Django Version: 1.4 pre-alpha SVN-16322
Python Version: 2.6.2
Installed Applications:
['django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.sites',
'django.contrib.messages',
'django.contrib.staticfiles',
'django.contrib.admin',
'emo20qBrowser.emotions']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware')
Traceback:
File "/home/abe/bin/django-trunk/django/core/handlers/base.py" in get_response
111. response = callback(request, *callback_args, **callback_kwargs)
File "/home/abe/bin/django-trunk/django/contrib/admin/options.py" in wrapper
316. return self.admin_site.admin_view(view)(*args, **kwargs)
File "/home/abe/bin/django-trunk/django/utils/decorators.py" in _wrapped_view
91. response = view_func(request, *args, **kwargs)
File "/home/abe/bin/django-trunk/django/views/decorators/cache.py" in _wrapped_view_func
77. response = view_func(request, *args, **kwargs)
File "/home/abe/bin/django-trunk/django/contrib/admin/sites.py" in inner
196. return view(request, *args, **kwargs)
File "/home/abe/bin/django-trunk/django/utils/decorators.py" in _wrapper
25. return bound_func(*args, **kwargs)
File "/home/abe/bin/django-trunk/django/utils/decorators.py" in _wrapped_view
91. response = view_func(request, *args, **kwargs)
File "/home/abe/bin/django-trunk/django/utils/decorators.py" in bound_func
21. return func(self, *args2, **kwargs2)
File "/home/abe/bin/django-trunk/django/db/transaction.py" in inner
211. return func(*args, **kwargs)
File "/home/abe/bin/django-trunk/django/contrib/admin/options.py" in add_view
871. if form.is_valid():
File "/home/abe/bin/django-trunk/django/forms/forms.py" in is_valid
121. return self.is_bound and not bool(self.errors)
File "/home/abe/bin/django-trunk/django/forms/forms.py" in _get_errors
112. self.full_clean()
File "/home/abe/bin/django-trunk/django/forms/forms.py" in full_clean
269. self._post_clean()
File "/home/abe/bin/django-trunk/django/forms/models.py" in _post_clean
331. self.instance.clean()
Exception Type: TypeError at /admin/emotions/answer/add/
Exception Value: 'unicode' object is not callable

Okay, I think I figured it out... I'm using a variable/column called "clean". Django's admin interface has a method called "clean()" also, which does some kind of validation. It appears that there was some kind of naming conflict so I changed the variable to name to "cleaned" and then to make sure that it knows what database field to use (I'm using a legacy/preexisting db), I added a db_column option:
cleaned = models.TextField(blank=True,db_column="clean")
It would have been nice to know that "clean" was a reserved identifier in django but at least I only wasted half a day on this django stuff which ostensibly makes database operations easier. To be fair, I just started django this morning and if I would have found and answer on stackoverflow it would have been a breeze to fix.
If anyone knows a better way to handle this, let me know...