I am using Django for an application that uses a simple filtering system. I want the filter to test if the title of my model contains a query string.
The code, stripped down, looks like this:
cards = Card.objects.filter(title__icontains=query)
print cards.query
which returns the following query (again, unnecessary stuff is stripped):
SELECT [...] FROM `ygo_card_card`
WHERE `ygo_card_card`.`title` LIKE %dark%
Which returns no results, even though it should. When I run this query manually, I get
SQL Error (1064): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%dark%' at line 1
If I wrap the %dark% part between apostrophes ('%dark%') when running manually, it works as expected. It seems to me that Django creates an incorrect query. Is this a bug or something that I can control by settings?
Any search returns irrelevant results, because the related keywords are too generic.
I use Django 1.6.5 and MySQL 5.5.38, running on Ubuntu Server 14.04 LTS.
The response is quite simple: I misinterpreted the problem.
The issue comes from an underlying problem: the MySQL LIKE statement is case-sensitive or insensitive depending on the collation and the Django filter used (icontains or contains) has no effect in the outcome. You can see this bug ticket for more information.
As Daniel Roseman pointed out, the .query property is not reliable, as the query is further processed by the database driver. This led me to believe that Django created a wrong query, while in fact it simply created a case-sensitive search that should have been case-insensitive, hence the lack of results.
In the end, the issue was resolved by changing the collation on columns, tables and the database.
Related
I am building an app with Django relying on several PosgreSQL databases which I do not manage, let's call them database A and database B. For each database, I've used python manage.py inspectdb to build my models.py files.
I am trying to use Django ORM to perform the following query (significantly simplified here), to_date being a datetime.datetime object:
sample = my_model_in_B.objects\
.filter(a_column=instance_of_a_model_in_A.name)\
.exclude(another_column='Useless things')\
.filter(a_column_with_dates__lte=to_date)
My issue is that it produces the following SQL query:
SELECT "myschema"."mytable"."a_column", "myschema"."mytable"."another_column" from "myschema"."mytable"
WHERE "myschema"."mytable"."a_column" = Here is the name of instance_of_a_model_in_A
AND "myschema"."mytable"."a_column_with_dates" <= 2020-02-03
AND NOT ("myschema"."mytable"."another_column" = Useless things
AND "myschema"."mytable"."another_column" IS NOT NULL))
In other terms, my issue is that the Django ORM does not automatically add quotes where I need them. I don't understand what I did wrong. I don't know if that matters but note that:
I use Django 2.2. and my database B is POSTGRESQL 8 only,
All the columns that I use correspond to CharField in my models, except a_column_with_dates which corresponds to a DateField.
Actually, my initial question was wrongly worded and misleading. I was assuming that QuerySet.query (that I used to get the above SQL code) was supposed to return the valid SQL query behind Django ORM, but it isn't. It just aims at providing a basic representation of the query, nonetheless, from a comment made on the official Django project website:
Django never actually interpolates the parameters: it sends the query
and the parameters separately to the database adapter, which performs
the appropriate operations.
Indeed, from the official documentation:
The query parameter to QuerySet exists so that specialized query subclasses can reconstruct internal query state. The value of the parameter is an opaque representation of that query state and is not part of a public API.
I'm facing a problem which I have no more ideas how to resolve.
I have need to test data which is returned by direct query from database.
During execution of TestCase django.db.connection.cursor() is returning data from main database, not from test one, which contain fixtures prepared for this test purposes.
I've tried to use both TestCase and TransactionalTestCase.
I've tried debugging, checking variables values and found out that connection is pointed onto test database for sure.
Do you know why it's returning data from main database? Is there any case when Django is copying data from main database to this created for tests purposes?
I'm using: Python 3.6.5, Django 2.1, pytest 4.6.3, pytest-django 3.5
Thanks in advance for any support.
Follow-up
Dears,
That problem occurs only while trying to perform custom raw SQL Query directly inside Test Case. Retrieving objects by standard Django QuerySets works fine.
Do you have any idea why this specific way to retrieve data from db not working during test execution?
I found an answer in Django documentation:
If your code attempts to access the database when its modules are compiled, this will occur before the test database is set up, with potentially unexpected results.
Still - if you know any way to avoid affecting tests by production data while raw sql queries are performed I would love to know how to do it.
I'm new to Flask and web development in general. I have a Flask web-application that is using SQLAlchemy, is it ok to put session.rollback at the beginning of the app in order to keep it running even after a transaction fails?
I had a problem with my website when it stopped working after I was attempting to delete records of one table. The error log showed that the deletion failed due to entries in another table still referencing these records as their foreign key. The error log suggested using session.rollback to rollback this change, so I put it at the beginning of my app just after binding my database and creating the session and my website worked. This gave me the hint to leave that line there. Is my move right, safe and ok? Can anyone tell me what is the correct thing to do if this is somewhat endangering the functionality or logic of my website by any chance?
I'd say by that you are by definition cargo cult coding and should try to determine why you're finding these errors in the first place instead of just including a bit of code for a reason you don't understand.
The problem you're describing is the result of using foreign keys to ensure data integrity in your database. Typically SQLAlchemy will nullify all of the depending foreign keys, but since I don't know anything about your set up I can't explain why it wasn't. It is perhaps a difference between databases.
One massive problem with putting the rollback at the beginning of a route (or the entire global app) is that you might rollback data which you didn't want to. You haven't provided an MVCE so no one can really help you debug your problem.
Cargo cult coding in circumstances like this is understandable, but it's never a good practice. To solve this problem, investigate the cascades in SQLAlchemy. Also, fire up your actual SQL db interface and look at the data's structure, and set SQLALCHEMY_ECHO = 1 in your config file to see what's actually getting emitted.
Good luck!
You should not use the rollback at the beginning but when a database operation fails.
The error is due to an integrity condition in your database. Some rows in your table are being referenced by another table. So, you have to remove referencing rows first.
Give it any basic model say;
class Post(models.Model):
created = models.DateTimeField(auto_now_add=True)
title = models.CharField(_('Title'), max_length=100)
content = models.TextField(_('Content html'), max_length=65000)
author = models.ForeignKey('user.User', on_delete=models.SET_NULL)
A query like Post.objects.annotate(Count('id')) (or any field, any annotate()) fails with the following error:
ProgrammingError: column "post.created" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: SELECT "post"."id", "post"."created", "post"."ti...
Using django 1.11.16, and postgres 9.4.19.
As I read here in another stackoverflow question tried different django versions and postgres versions; using django 2.0, 2.1.2, postgres 9.5.. same error! Reading around I've seen that might be a problem related to SQL, but I'm having this problem only at one server running Ubuntu 18.04 (bionic). Running the query locally at Ubuntu 16.04, with django 1.11.16, or any version above that and postgres 9.4 or above runs fine in my local system.. So the problem might actually be related with some low level libraries perhaps, I'm not running complex queries, any simple annotate() with django 1.11+ fails in ubuntu 18.04 with postgres 9.4 or 9.5
[UPDATE]
Might be useful for you if you find yourself in this scenario with no clues of what's going on, verify that the table in question has its indexes created. My problem turned out to be the posts table had no PRIMARY KEY definition nor any other constraint, a fail in a pg_restore which restored all the data and some schema definitions, yeah you read right some other schema definitions were missing, no idea of why.. But instead of trying to debug what happened with pg_restore in the first place, I ran an initial python manage migrate on an empty DB so the schema was correctly created this time and verified (psql -d <db_name> -c '\d posts'), then run pg_restore again with the --data-only and --disable-triggers flags.. So finally I got the schema and data properly restored and the query worked
That error message is because PostgreSQL will not make a guess as to what to do with non-grouped columns when there's a aggregate function in the query. This is one of the cases where the Django ORM handwaves too much and allows us to shoot ourselves in the foot.
I ran a test on my projects that use Django 2.1 and another that uses 1.11 with Model.objects.annotate(Count('id')) with no issues. If you post the full QuerySet, people will be able to help you further.
I met the problem and I solved it.
you should add the .order_by("xxx") follow your ORM!
before:
search_models.Author.objects.values("first_institution", 'institution_code').annotate(counts=Count('first_institution'))
after:
search_models.Author.objects.values("first_institution", 'institution_code').annotate(counts=Count('first_institution')).order_by('-counts')
It's worked for me, also wish can help you.
I'm working on a script to automate a file load procedure. So, naturally I need to perform some stored procedures that already exist. I'm using pyodbc to connect to my database. I can SELECT perfectly fine from the database, but when I try to execute from the database I get this error:
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][SQL Server Native Client 10.0]
Syntax error, permission violation, or other nonspecific error (0) (SQLExecDirectW)')
I can't figure out what the problem here is - the user has full DB admin permissions, the syntax is correct based off what the pyodbc official documentation says.
print("Executing SP")
conn.execute('{EXEC TEMP.s_p_test}')
print("SP Executed.")
Here, TEMP is the schema for the type of stored procedure in that specific database. I.e., it's the full name of the stored procedure. I feel like it's probably something stupidly obvious that I'm just missing.
I tried a couple of things to fix it. As #Brian Pendleton suggested, I had tried to change from an explicit database user defined via UID and PWD to trusted_connection=True. Unfortunately that did not change anything.
However, out of curiosity I decided to see what taking the curly braces out of the function call would do. The execution worked immediately and produced the desired output. It would seem that the documentation at pyodbc's wiki either shows bad examples or I found a bug I don't know how to replicate because I don't know what makes my situation abnormal.
Or, in other words, instead of
conn.execute('{EXEC TEMP.s_p_test}')
I used
conn.execute('EXEC TEMP.s_p_test')