We Keep Coding

django custom Func for specific SQL function

I'm currently performing a raw query in my database because i use the MySQL function instr. I would like to translate is into a django Func class.I've spend several days reading the docs, Django custom for complex Func (sql function) and Annotate SQL Function to Django ORM Queryset `FUNC` / `AGGREGATE` but i still fail to write succesfully my custom Func.
This is my database
from django.db import models
class Car(models.Model):
brand = models.CharField("brand name", max_length=50)
#then add the __str__ function
Then I populate my database for this test
Car.objects.create(brand="mercedes")
Car.objects.create(brand="bmw")
Car.objects.create(brand="audi")
I want to check if something in my table is in my user input. This is how i perform my SQL query currently
query = Car.objects.raw("SELECT * FROM myAppName_car WHERE instr(%s, brand)>0", ["my audi A3"])
# this return an sql query with one element in this example
I'm trying to tranform it in something that would look like this
from django.db.models import Func
class Instr(Func):
function = 'INSTR'
query = Car.objects.filter(brand=Instr('brand'))
EDIT
Thank to the response, the correct answer is
from django.db.models import Value
from django.db.models.functions import StrIndex
query = Car.objects.annotate(pos=StrIndex(Value("my audi A3"), "brand")).filter(pos__gt=0)

Your custom database function is totally correct, but you're using it in the wrong way.
When you look at your usage in the raw SQL function, you can clearly see that you need 3 parameters for your filtering to work correctly: a string, a column name and a threshold (in your case it is always zero)
If you want to use this function in the same way in your query, you should do it like this:
query = Car.objects.annotate(
position_in_brand=Instr("my audi A3")
).filter(position_in_brand__gt=0)
For this to work properly, you also need to add output_field = IntegerField() to your function class, so Django knows that the result will always be an integer.
But... You've missed 2 things and your function is not needed at all, because it already exist in Django as StrIndex.
And the 2nd thing is: you can achieve the same outcome by using already existing django lookups, like contains or icontains:
query = Car.objects.filter(brand__contains="my audi A3")
You can find out more about lookups in Django docs

Why .query() function not working on Django ORM query?

As I already know that using .query.__str__() , we can get sql equivalent query from Django ORM query.
e.g : Employees.objects.filter(id = int(id)).query.__str__()
Above code working well & I am able to get sql equivalent query
but when I am using same on below query I am getting error like below.
Employees.objects.filter(id = int(id)).first().query.__str__()
AttributeError: 'Employees' object has no attribute 'query'
Why now I am getting error, any suggestions ?

.first() [Django-doc] does not return a QuerySet, it returns a model object. The query is evaluated eagerly.
You can inspect the last query that Django made with:
from django.db import connection
print(connection.queries[-1:])
That being said, in essence a some_queryset.first() is often the same query as some_queryset, except that it will limit the queryset.
Note: Please do not use .__str__, you can use str(my_queryset.query), or just print(my_queryset.query).

Django ManyToMany through with multiple databases

TLTR: Django does not include database names in SQL queries, can I somehow force it to do this or is there a workaround?
The long version:
I have two legacy MySQL databases (Note: I have no influence on the DB layout) for which I'm creating a readonly API using DRF on Django 1.11 and python 3.6
I'm working around the referential integrity limitation of MyISAM DBs by using the SpanningForeignKey field suggested here: https://stackoverflow.com/a/32078727/7933618
I'm trying to connect a table from DB1 to a table from DB2 via a ManyToMany through table on DB1. That's the query Django is creating:
SELECT "table_b"."id" FROM "table_b" INNER JOIN "throughtable" ON ("table_b"."id" = "throughtable"."b_id") WHERE "throughtable"."b_id" = 12345
Which of course gives me an Error "Table 'DB2.throughtable' doesn't exist" because throughtable is on DB1 and I have no idea how to force Django to prefix the tables with the DB name. The query should be:
SELECT table_b.id FROM DB2.table_b INNER JOIN DB1.throughtable ON (table_b.id = throughtable.b_id) WHERE throughtable.b_id = 12345
Models for app1 db1_app/models.py: (DB1)
class TableA(models.Model):
id = models.AutoField(primary_key=True)
# some other fields
relations = models.ManyToManyField(TableB, through='Throughtable')
class Throughtable(models.Model):
id = models.AutoField(primary_key=True)
a_id = models.ForeignKey(TableA, to_field='id')
b_id = SpanningForeignKey(TableB, db_constraint=False, to_field='id')
Models for app2 db2_app/models.py: (DB2)
class TableB(models.Model):
id = models.AutoField(primary_key=True)
# some other fields
Database router:
def db_for_read(self, model, **hints):
if model._meta.app_label == 'db1_app':
return 'DB1'
if model._meta.app_label == 'db2_app':
return 'DB2'
return None
Can I force Django to include the database name in the query? Or is there any workaround for this?

A solution exists for Django 1.6+ (including 1.11) for MySQL and sqlite backends, by option ForeignKey.db_constraint=False and explicit Meta.db_table. If the database name and table name are quoted by ' ` ' (for MySQL) or by ' " ' (for other db), e.g. db_table = '"db2"."table2"'). Then it is not quoted more and the dot is out of quoted. Valid queries are compiled by Django ORM. A better similar solution is db_table = 'db2"."table2' (that allows not only joins but it is also by one issue nearer to cross db constraint migration)
db2_name = settings.DATABASES['db2']['NAME']
class Table1(models.Model):
fk = models.ForeignKey('Table2', on_delete=models.DO_NOTHING, db_constraint=False)
class Table2(models.Model):
name = models.CharField(max_length=10)
....
class Meta:
db_table = '`%s`.`table2`' % db2_name # for MySQL
# db_table = '"db2"."table2"' # for all other backends
managed = False
Query set:
>>> qs = Table2.objects.all()
>>> str(qs.query)
'SELECT "DB2"."table2"."id" FROM DB2"."table2"'
>>> qs = Table1.objects.filter(fk__name='B')
>>> str(qs.query)
SELECT "app_table1"."id"
FROM "app_table1"
INNER JOIN "db2"."app_table2" ON ( "app_table1"."fk_id" = "db2"."app_table2"."id" )
WHERE "db2"."app_table2"."b" = 'B'
That query parsing is supported by all db backends in Django, however other necessary steps must be discussed individually by backends. I'm trying to answer more generally because I found a similar important question.
The option 'db_constraint' is necessary for migrations, because Django can not create the reference integrity constraint
ADD foreign key table1(fk_id) REFERENCES db2.table2(id),
but it can be created manually for MySQL.
A question for particular backends is if another database can be connected to the default at run-time and if a cross database foreign key is supported. These models are also writable. The indirectly connected database should be used as a legacy database with managed=False (because only one table django_migrations for migrations tracking is created only in the directly connected database. This table should describe only tables in the same database.) Indexes for foreign keys can however be created automatically on the managed side if the database system supports such indexes.
Sqlite3: It has to be attached to another default sqlite3 database at run-time (answer SQLite - How do you join tables from different databases), at best by the signal connection_created:
from django.db.backends.signals import connection_created
def signal_handler(sender, connection, **kwargs):
if connection.alias == 'default' and connection.vendor == 'sqlite':
cur = connection.cursor()
cur.execute("attach '%s' as db2" % db2_name)
# cur.execute("PRAGMA foreign_keys = ON") # optional
connection_created.connect(signal_handler)
Then it doesn't need a database router of course and a normal django...ForeignKey can be used with db_constraint=False. An advantage is that "db_table" is not necessary if the table names are unique between databases.
In MySQL foreign keys between different databases are easy. All commands like SELECT, INSERT, DELETE support any database names without attaching them previously.
This question was about legacy databases. I have however some interesting results also with migrations.

I have a similar setup with PostgreSQL. Utilizing search_path to make cross-schema references possible in Django (schema in postgres = database in mysql). Unfortunately, seems like MySQL doesn't have such a mechanism.
However, you might try your luck creating views for it. Make views in one databases that references other databases, use it to select data. I think it's the best option since you want your data read-only anyway.
It's however not a perfect solution, executing raw queries might be more useful in some cases.
UPD: Providing mode details about my setup with PostgreSQL (as requested by bounty later). I couldn't find anything like search_path in MySQL documentation.
Quick intro
PostgreSQL has Schemas. They are synonymous to MySQL databases. So if you are MySQL user, imaginatively replace word "schema" with word "database". Requests can join tables between schemas, create foreign keys, etc... Each user (role) has a search_path:
This variable [search_path] specifies the order in which schemas are searched when
an object (table, data type, function, etc.) is referenced by a simple
name with no schema specified.
Special attention on "no schema specified", because that's exactly what Django does.
Example: Legacy databases
Let's say we got coupe legacy schemas, and since we are not allowed to modify them, we also want one new schema to store the NM relation in it.
old1 is the first legacy schema, it has old1_table (which is also the model name, for convenience sake)
old2 is the second legacy schema, it has old2_table
django_schema is a new one, it will store the required NM relation
All we need to do is:
alter role django_user set search_path = django_schema, old1, old2;
This is it. Yes, that simple. Django has no names of the schemas ("databases") specified anywhere. Django actually has no idea what is going on, everything is managed by PostgreSQL behind the scenes. Since django_schema is first in the list, new tables will be created there. So the following code ->
class Throughtable(models.Model):
a_id = models.ForeignKey('old1_table', ...)
b_id = models.ForeignKey('old2_table', ...)
-> will result in a migration that creates table throughtable that references old1_table and old2_table.
Problems: if you happened to have several tables with same names, you will either need to rename them or still trick Django into using a dot inside of table names.

Django does have the ability to work with multiple databases. See https://docs.djangoproject.com/en/1.11/topics/db/multi-db/.
You can also use raw SQL queries in Django. See https://docs.djangoproject.com/en/1.11/topics/db/sql/.

updating a field in model according to count "many to many" field

I have a django model called Friend that contains a many to many field called friends through another model called FriendshipInfo. For performance reasons i decided to hold a field that holds the number of friends each person has. So now in the migration scripts i need to update the existing friends in my db. This is how i did it:
def forward(...):
# ... adding the new count field ...
for person in Friend.objects.all():
person.friends_count = len(persion.friends.all())
person.uupdate()
I was wondering if there is any way to do this in a much more efficient way (bulk update somehow?)
Tech info:
I am using Python 2.7
I am using django 1.6
For migrations I'm using south

I was tempted to use the extra queryset method to grab the count of friends and bulk update your Friend objects like:
def forward(...):
# adding the new count field
Friend.objects.extra(select = {
'friends_number': 'SELECT COUNT(*) FROM <your_many_to_many_table_name> WHERE <your_many_to_many_table_name>.<your_FriendshipInfo_related_name> = <your_Friend_table_name>.id'
}).update(friends_count=F('friends_number'))
But, by the look of things, it is not possible. However, you can use the raw method custom SQL queries with an update from count query:
from django.db import connection
def forward(...):
# adding the new count field
cursor = connection.cursor()
cursor.execute('\
UPDATE <your_Friend_table_name>\
SET friends_count = \
(SELECT COUNT(*) FROM <your_many_to_many_table_name> WHERE <your_many_to_many_table_name>.<your_FriendshipInfo_related_name> = <your_Friend_table_name>.id)')

the right way it's in the migration file do the data migration (.py)
there you can put the sql query without a problem, the method is migrations.RUNSQL
here it's a example:
class Migration(migrations.Migration):
dependencies = [
('procesos', '0020_auto_20150703_1656'),
]
operations = [
migrations.RunSQL("UPDATE procesos_busquedainmueble SET tipo_inmueble_id=(filtros::json->>'tipo_inmueble')::int;"),

Checking for non-existent optional tables

I am using SQLAlchemy + Pyramid to operate on my database. However, there are some optional tables which are not always expected to be present in the DB. So while querying them I try to catch such cases with the NoSuchTableError
try:
x = session.query(ABC.name.label('sig_name'),func.count('*').label('count_')).join(DEF).join(MNO).filter(MNO.relevance >= relevance_threshold).group_by(DEF.signature).order_by(desc('count_')).all()[:val]
except NoSuchTableError:
x = [-1,]
But on executing this statement, I get a ProgrammingError
ProgrammingError: (ProgrammingError) (1146, "Table 'db.mno' doesn't exist")
Why does SQLAlchemy raise the more general ProgrammingError instead of the more specific NoSuchTableError? And if this is indeed expected behaviour, how do I ensure the app displays correct information depending on whether tables are present/absent?
EDIT1
Since this is part of my webapp, the model of DB is in models.py (under my pyramid webapp). I do have a setting in my .ini file that asks user to select whether additional tables are available or not. But not trusting the user, I want to be able to check for myself (in the views) whether table exists or not. The contentious table is something like (in models.py)
class MNO(Base):
__tablename__="mno"
id=Column(Integer,primary_key=True,autoincrement=True)
sid=Column(Integer)
cid=Column(mysql.MSInteger(unsigned=True))
affectability=Column(Integer)
cvss_base=Column(Float)
relevance=Column(Float)
__table_args__=(ForeignKeyConstraint(['sid','cid',],['def.sid','def.cid',]),UniqueConstraint('sid','cid'),)
How and Where should the check be made so that a variable can be set (preferably during app setup) which tells me whether the tables are present or not?
Note: In this case I would have to try if...else rather than 'ask for forgiveness'

According to the sqlalchemy docs, a NoSuchTableError is only thrown when "SQLAlchemy [is] asked to load a table's definition from the database, but the table doesn't exist." You could try loading a table's definition, catching the error there, and doing your query otherwise.
If you want to do things via "asking for forgiveness":
try:
table = Table(table_name, MetaData(engine))
except NoSuchTableError:
pass
Alternatively, you could just check whether the table exists:
Edit:
Better yet, why don't you use the has_table method:
if engine.dialect.has_table(connection, table_name):
#do your crazy query
Why don't you use Inspector to grab the table names first?
Maybe something like this:
from sqlalchemy import create_engine
from sqlalchemy.engine import reflection
#whatever code you already have
engine = create_engine('...')
insp = reflection.Inspector.from_engine(engine)
table_name = 'foo'
table_names = insp.get_table_names()
if table_name in table_names:
x = session.query(ABC.name.label('sig_name'),func.count('*').label('count_')).join(DEF).join(MNO).filter(MNO.relevance >= relevance_threshold).group_by(DEF.signature).order_by(desc('count_')).all()[:val]