Django ORM - update() on rows multiplied with Decimal() - python

I need to execute a query like UPDATE sometable SET total_cost=cost*sec/60.0 via Django ORM. Query will produce some warnings similar to problems described in this ticket.
new_records = Service.objects.filter(upd='U')
print new_records.update(
total_cost=F('cost')*F('billsec')/Decimal('60.0')
)
(cost && sec fields are of DECIMAL(10,4) type).The result:
Traceback (most recent call last):
File "./test.py", line 15, in <module>
total_cost=F('cost')*F('billsec')/Decimal('60.0')
File "/usr/lib/python2.7/dist-packages/django/db/models/query.py", line 533, in update
rows = query.get_compiler(self.db).execute_sql(None)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 986, in execute_sql
cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
File "/usr/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 818, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/python2.7/dist-packages/django/db/backends/util.py", line 40, in execute
return self.cursor.execute(sql, params)
File "/usr/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 114, in execute
return self.cursor.execute(query, args)
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 176, in execute
if not self._defer_warnings: self._warning_check()
File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 92, in _warning_check
warn(w[-1], self.Warning, 3)
_mysql_exceptions.Warning: Data truncated for column 'total_cost' at row 1
If i'll drop Decimal() from filter query, it will work fine.

See this Django ticket
https://code.djangoproject.com/ticket/13666
It's said there that this is the problem of MySQL mishandling decimal string. Somebody said that casting decimal to float, i.e. float(Decimal('60.0')) should solve the issue.

Related

Django Make Migration Error No Such Column

I have a pre-existing model that I am now trying to add an additional field too. It wasn't me that made the original field but I have been adding fields to other classes and making migrations fine.
I want to add an additional image field to the class Event, there is already an image field on this class so I know it can handle image fields, I have tried changing the name around to see if that was a issue too. Here is the field I want to add:
social_media_image = models.ImageField(null=True, blank=True, help_text='Hello World')
This is the error I get when i'm trying to make my migration after adding that code to the model:
django.db.utils.OperationalError: no such column: posts_event.social_media_image
From my understanding of how migrations work there shouldn't be a column called this yet as I haven't made the migration yet that will add it to the DB.
I have tried adding a default value to the field as well but with no luck. I have also tried completely removing the DB along with migration files and trying to recreate them.
Here are the rest of the fields in the model:
slug = AutoSlugField(max_length=50, unique=True, populate_from='title')
content = models.TextField(default='')
start_time = models.DateTimeField()
image = models.ImageField(null=True, blank=True, help_text=image_help_text)
Here is the traceback:
Traceback (most recent call last):
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/db/backends/sqlite3/base.py", line 337, in execute
return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such column: posts_event.social_media_image
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "manage.py", line 22, in <module>
execute_from_command_line(sys.argv)
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
utility.execute()
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/core/management/__init__.py", line 341, in execute
django.setup()
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/__init__.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/apps/registry.py", line 115, in populate
app_config.ready()
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/source/enfield_presents/posts/apps.py", line 37, in ready
search.register(Event.objects.get_upcoming_events(site_id=settings.SITE_ID, include_spektrix=False, return_queryset=True))
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/source/enfield_presents/posts/models.py", line 282, in get_upcoming_events
events_with_a_next_start_time = [e for e in events if e.next_start_time() is not None]
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/db/models/query.py", line 256, in __iter__
self._fetch_all()
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/db/models/query.py", line 1087, in _fetch_all
self._result_cache = list(self.iterator())
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/db/models/query.py", line 54, in __iter__
results = compiler.execute_sql()
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
cursor.execute(sql, params)
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/db/backends/utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/db/utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/utils/six.py", line 685, in reraise
raise value.with_traceback(tb)
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/venv/lib/python3.4/site-packages/django/db/backends/sqlite3/base.py", line 337, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such column: posts_event.social_media_image
This issue may be due to the reason that the corresponding Image field may not be present in the database. Please login to the database shell and check whether the corresponding field is there. If it is not there add it manually using SQL query or rollback to a previous safe migration and then make the changes in models.py and generate the migration files.
It's right there:
File "/Applications/XAMPP/xamppfiles/htdocs/enfield/enfield-presents/source/enfield_presents/posts/apps.py", line 37, in ready:
search.register(Event.objects.get_upcoming_events(site_id=settings.SITE_ID, include_spektrix=False, return_queryset=True))
You do a queryset on the source of one app, that as far as I can tell, queries the model of the other app. You shouldn't do queries in app setup if you can avoid it. Use signals if you have to or lazy loading.
If you really have to, then flip the order of the apps. If it's the same app, then yes, don't do it: as you notice, you won't be able to do any migrations.

CKAN paster db init sqlalchemy programming error

I have followed the CKAN install guide from source (http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html) and managed to get all the way until point 6. Create database tables.
At that stage, running paster db init -c /etc/ckan/default/development.ini gives the following stack trace:
Traceback (most recent call last):
File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
result = self.command()
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 208, in command
self._load_config(cmd!='upgrade')
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 164, in _load_config
self.site_user = logic.get_action('get_site_user')({'ignore_auth': True}, {})
File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line 424, in wrapped
result = _action(context, data_dict, **kw)
File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/get.py", line 2209, in get_site_user
user = model.User.get(site_id)
File "/usr/lib/ckan/default/src/ckan/ckan/model/user.py", line 64, in get
return query.first()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2334, in first
ret = list(self[0:1])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2201, in __getitem__
return list(res)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2405, in __iter__
return self._execute_and_instances(context)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2420, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 727, in execute
return meth(self, multiparams, params)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 824, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 954, in _execute_context
context)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1116, in _handle_dbapi_exception
exc_info
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 189, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 947, in _execute_context
context)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 435, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column user.password does not exist
LINE 1: SELECT "user".password AS user_password, "user".id AS user_i...
^
'SELECT "user".password AS user_password, "user".id AS user_id, "user".name AS user_name, "user".openid AS user_openid, "user".fullname AS user_fullname, "user".email AS user_email, "user".apikey AS user_apikey, "user".created AS user_created, "user".reset_key AS user_reset_key, "user".about AS user_about, "user".activity_streams_email_notifications AS user_activity_streams_email_notifications, "user".sysadmin AS user_sysadmin, "user".state AS user_state \nFROM "user" \nWHERE "user".name = %(name_1)s OR "user".openid = %(openid_1)s OR "user".id = %(id_1)s ORDER BY "user".name \n LIMIT %(param_1)s' {'param_1': 1, 'id_1': 'default', 'name_1': 'default', 'openid_1': 'default'}
I did not make any modifications to the installation or the schema. The installation is under a proxy, but that did not turn out to be a problem thus far, since there were many posts giving help in that regard.
Did anybody come across this error before and managed to resolve it? I don't suppose it is a proxy-related problem... though it seems to be related to the schema.
I ran into the same issue but dropping the DB and creating and initialising it again afterwards produced the same error. Also running paster db clear -c /etc/ckan/default/development.ini failed with said error.
It worked for me, when I deactivated all ckan.plugins in development.ini before running paster db init -c /etc/ckan/default/development.ini.
Managed to solve this one after all, so I'll answer my own question... :)
The problem was with the setup of the database. Once I purged the database
and completely reinstalled it, everything worked like a charm!
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column user.password does not exist
LINE 1: SELECT "user".password AS user_password, "user".id AS user_i...
This error usually means that the CKAN database exists but is empty - the CKAN tables have not been created in it.
To solve it:
cd /usr/lib/ckan/default/src/ckan
paster db init -c /etc/ckan/default/development.ini
There's more about this in the docs:
http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html#create-database-tables

LOAD DATA LOCAL INFILE sqlalchemy and python to mysql db

I'm trying to use LOAD LOCAL DATA INFILE with sqlalchemy to upload batch files to my server. I have edited my /etc/mysql/my.conf to contain local-infile == 1 under both [mysql] and [mysqld], yet I'm getting this traceback when I run the program.
The relevant line of code is:
Traceback (most recent call last):
File "main.py", line 48, in <module>
con.execute(sql)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 906, in execute
return self._execute_text(object, multiparams, params)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1054, in _execute_text
statement, parameters
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1332, in _handle_dbapi_exception
exc_info
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 442, in do_execute
cursor.execute(statement, parameters)
File "/Users/eoddata/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/Users/eoddata/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1148, 'The used command is not allowed with
this MySQL version') [SQL: "LOAD DATA LOCAL
INFILE 'stocks/AMEX_20150420.txt' INTO TABLE database.new_table\n
FIELDS TERMINATED BY ',' (Symbol,#Date,Open,High,Low,Close,Volume)\n
SET Date = STR_TO_DATE(#Date,'%%Y%%m%%d');"]
I've done some research from the docs and can't find a way to fix this error. With python MySQLdb (from pip install MySQL-Python) will allow me to pass in a local-infile paramter. I can't find an equivalent in SQLAlchemy. Any help is appreciated.
EDIT: I've worked on this all night and still haven't made any progress - a lot of dead ends. Any ideas?
Thanks, Jared
You should add local_infile=1 to the connection string.
SQLALCHEMY_DATABASE_URI = "mysql://user:pass#localhost/dbname?charset=utf8&local_infile=1"

sqlalchemy with postgres: insert into a table whose columns have parentheses

Suppose you have a table "foo" in postgres with column name "col (parens) name". The psql command
INSERT INTO "foo" ("col (parens) name") VALUES ('bar');
works just fine. However, if I try to do the same using sqlalchemy (version 0.9.7), the resulting python code fails:
conn = sqlalchemy.create_engine('postgresql://name:password#host:port/database')
meta = sqlalchemy.schema.MetaData()
meta.reflect(bind=conn)
foo = meta.tables['foo']
vals = [{'col (parens) name': 'hi'}, {'col (parens) name': 'bye'}]
conn.execute(foo.insert(values=vals))
This does not work, giving the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sqlalchemy/engine/base.py", line 729, in execute
return meth(self, multiparams, params)
File "sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
compiled_sql, distilled_params
File "sqlalchemy/engine/base.py", line 957, in _execute_context
context)
File "sqlalchemy/engine/base.py", line 1162, in _handle_dbapi_exception
util.reraise(*exc_info)
File "sqlalchemy/engine/base.py", line 950, in _execute_context
context)
File "sqlalchemy/engine/default.py", line 436, in do_execute
cursor.execute(statement, parameters)
KeyError: 'col (parens'
Apparently the sqlalchemy method to bind db parameters is running into trouble with python string interpolation. Any suggestions for a workaround?
Add paramstyle="format" to the create_engine call. It will change the way the query values are inserted to the query in a way that it won't crash on closing brackets.
conn = sqlalchemy.create_engine(
'postgresql://name:password#host:port/database',
paramstyle="format"
)

silently truncating django model char field

I'm trying to do a mass import of data into an existing database. Sometimes the string fields in this other database contain strings that are larger than the field I'm importing them into. I'm OK with this, I'd like the system to just silently truncate, but I'm getting a warning which then also halts execution.
Since this is a live database, I'm running my code via "manage.py shell" and and when I call the import routine it halts after the first string field it encounters that's too long. Oddly it DOES insert it into the database truncated as I expect, but then exits the for loop.
Thanks!
Here's the error output:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/blah/blah/update.py", line 41, in updateCommunities
community.save()
File "/usr/lib/pymodules/python2.6/django/db/models/base.py", line 434, in save
self.save_base(using=using, force_insert=force_insert, force_update=force_update)
File "/usr/lib/pymodules/python2.6/django/db/models/base.py", line 527, in save_base
result = manager._insert(values, return_id=update_pk, using=using)
File "/usr/lib/pymodules/python2.6/django/db/models/manager.py", line 195, in _insert
return insert_query(self.model, values, **kwargs)
File "/usr/lib/pymodules/python2.6/django/db/models/query.py", line 1479, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/lib/pymodules/python2.6/django/db/models/sql/compiler.py", line 783, in execute_sql
cursor = super(SQLInsertCompiler, self).execute_sql(None)
File "/usr/lib/pymodules/python2.6/django/db/models/sql/compiler.py", line 727, in execute_sql
cursor.execute(sql, params)
File "/usr/lib/pymodules/python2.6/django/db/backends/util.py", line 15, in execute
return self.cursor.execute(sql, params)
File "/usr/lib/pymodules/python2.6/django/db/backends/mysql/base.py", line 86, in execute
return self.cursor.execute(query, args)
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 168, in execute
if not self._defer_warnings: self._warning_check()
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 82, in _warning_check
warn(w[-1], self.Warning, 3)
Warning: Data truncated for column 'community_name' at row 1

Categories