We Keep Coding

How do I design my django app which fetches the data from a legacy database which has huge number of tables?

I am developing a Django app which fetches the data from a huge database and then perform some statistical operation and prints the result. There are different ways available but I want to know what is the best way of doing it?
Dataset have a lot of tables, so is it advisable to create models for all those tables? and use Django ORM to replicate below query?
OR else I should not use Models at all and use RAW SQL query to fetch the data from database and perform operation and then show the result. If yes then please suggest the best way of doing it.
"USE r_d;
SELECT nus.noyau_utilisateur_id as "candidate_id",
pcon.id as "assessment_id",
pcon.created_at as "assessment_creation_date",
YEAR(pcon.created_at) as "assessment_year",
tti.libelle as "test_name",
tfi.libelle as "factor_name",
pcal.score_reel as "real_score",
pcal.score_brut as "raw_score",
pcal.score_calcule as "calculated_score"
FROM `passation_calcul` pcal
JOIN `passation_contrat` pcon ON pcon.id = pcal.passation_contrat_id
JOIN `noyau_utilisateur_societe` nus ON nus.id = pcon.noyau_utilisateur_societe_id
JOIN `test_test` tt ON tt.id = pcon.test_test_id
JOIN `test_test_i18n` tti ON tt.id = tti.test_test_id
JOIN `test_facteur` tf ON tf.id = pcal.id_ref
JOIN `test_facteur_i18n` tfi ON tf.id = tfi.test_facteur_id
WHERE pcon.test_test_id = 144
AND pcon.etat = "COMPLETED"
AND tti.noyau_ref_langue_produit_id = 1
AND pcal.table_ref = 'test_facteur'
AND tfi.noyau_ref_langue_produit_id = 1
ORDER BY candidate_id, assessment_id, factor_name"```

I think you, after specifying the database settings to point to your legacy db in the settings.py, you should run
python manage.py inspectdb
that will generate python model classes of the existing table isn the db.. write the output to models.py and chek it out
This might help. more detailed.
https://django-book.readthedocs.io/en/latest/chapter18.html

After copying the output to models.py.. make the required changes if any.
python manage.pymakemigrations
Now we need to migrate, that will create the required tables in the db in accordance to the models in model.py.
But we already have those tables.. that's where fake migration comes in..
python manage.py migrate --fake
Before starting read this.. and understand what you are doing.. this will enable you to make django orm queries without using SQL seamlessly.
https://docs.djangoproject.com/en/4.0/ref/django-admin/#migrate

How to join tables from two different databases using sqlalchemy expression language / sqlalchemy core?

I am using MySql. I was however able to find ways to do this using sqlalchemy orm but not using expression language.So I am specifically looking for core / expression language based solutions. The databases lie on the same server
This is how my connection looks like:
connection = engine.connect().execution_options(
schema_translate_map = {current_database_schema: new_database_schema})
engine_1=create_engine("mysql+mysqldb://root:user#*******/DB_1")
engine_2=create_engine("mysql+mysqldb://root:user#*******/DB_2",pool_size=5)
metadata_1=MetaData(engine_1)
metadata_2=MetaData(engine_2)
metadata.reflect(bind=engine_1)
metadata.reflect(bind=engine_2)
table_1=metadata_1.tables['table_1']
table_2=metadata_2.tables['table_2']
query=select([table_1.c.name,table_2.c.name]).select_from(join(table_2,table_1.c.id==table_2.c.id,'left')
result=connection.execute(query).fetchall()
However, when I try to join tables from different databases it throws an error obviously because the connection belongs to one of the databases. And I haven't tried anything else because I could not find a way to solve this.
Another way to put the question (maybe) is 'how to connect to multiple databases using a single connection in sqlalchemy core'.

Applying the solution from here to Core only you could create a single Engine object that connects to your server, but without defaulting to one database or the other:
engine = create_engine("mysql+mysqldb://root:user#*******/")
and then using a single MetaData instance reflect the contents of each schema:
metadata = MetaData(engine)
metadata.reflect(schema='DB_1')
metadata.reflect(schema='DB_2')
# Note: use the fully qualified names as keys
table_1 = metadata.tables['DB_1.table_1']
table_2 = metadata.tables['DB_2.table_2']
You could also use one of the databases as the "default" and pass it in the URL. In that case you would reflect tables from that database as usual and pass the schema= keyword argument only when reflecting the other database.
Use the created engine to perform the query:
query = select([table_1.c.name, table_2.c.name]).\
select_from(outerjoin(table1, table_2, table_1.c.id == table_2.c.id))
with engine.begin() as connection:
result = connection.execute(query).fetchall()

select in (select ..) using ORM django

How can I make a query
select name where id in (select id from ...)
using Django ORM? I think I can make this using some loop for for obtain some result and another loop for, for use this result, but I think that is not practical job, is more simple make a query sql, I think that make this in python should be more simple in python
I have these models:
class Invoice (models.Model):
factura_id = models.IntegerField(unique=True)
created_date = models.DateTimeField()
store_id = models.ForeignKey(Store,blank=False)
class invoicePayments(models.Model):
invoice = models.ForeignKey(Factura)
date = models.DateTimeField()#auto_now = True)
money = models.DecimalField(max_digits=9,decimal_places=0)
I need get the payments of a invoice filter by store_id,date of pay.
I make this query in mysql using a select in (select ...). This a simple query but make some similar using django orm i only think and make some loop for but I don't like this idea:
invoiceXstore = invoice.objects.filter(local=3)
for a in invoiceXstore:
payments = invoicePayments.objects.filter(invoice=a.id,
date__range=["2016-05-01", "2016-05-06"])

You can traverse ForeignKey relations using double underscores (__) in Django ORM. For example, your query could be implemented as:
payments = invoicePayments.objects.filter(invoice__store_id=3,
date__range=["2016-05-01", "2016-05-06"])
I guess you renamed your classes to English before posting here. In this case, you may need to change the first part to factura__local=3.
As a side note, it is recommended to rename your model class to InvoicePayments (with a capital I) to be more compliant with PEP8.

Your mysql raw query is a sub query.
select name where id in (select id from ...)
In mysql this will usually be slower than an INNER JOIN (refer : [http://dev.mysql.com/doc/refman/5.7/en/rewriting-subqueries.html]) thus you can rewrite your raw query as an INNER JOIN which will look like 1.
SELECT ip.* FROM invoicepayments i INNER JOIN invoice i ON
ip.invoice_id = i.id
You can then use a WHERE clause to apply the filtering.
The looping query approach you have tried does work but it is not recommended because it results in a large number of queries being executed. Instead you can do.
InvoicePayments.objects.filter(invoice__local=3,
date__range=("2016-05-01", "2016-05-06"))
I am not quite sure what 'local' stands for because your model does not show any field like that. Please update your model with the correct field or edit the query as appropriate.
To lean about __range see this https://docs.djangoproject.com/en/1.9/ref/models/querysets/#range

How do I do a "starts with" query using SQL alchemy?

I am learning to use SQL alchemy to connect to a mysql database. I want to pull records from the DB that start with a given string. I know that for simple equality all I need to do is this
queryRes = ses.query(Table).filter(Table.fullFilePath == filePath).all()
result = []
How do I do something like this?
queryRes = ses.query(Table).filter(Table.fullFilePath.startsWith(filePath)).all()
result = []
Maybe the query would look like this?
q = ses.query(Table).filter(Table.fullFilePath.like('path%')).all()

SQLAlchemy has a startswith column property, so it works exactly as you'd think:
queryRes = ses.query(Table).filter(Table.fullFilePath.startswith(filePath)).all()

This is the pure SQL:
SELECT * FROM table WHERE field LIKE "string%"
The SQL alchemy is:
q = ses.query(Table).filter(Table.fullFilePath.like('path%')).all()

If you need a case insensitive comparison, use ilike:
session.query(SomeTable).filter(SomeTable.some_column.ilike('bla%')).all()

django/python: raw sql with multiple tables

I need to perform a raw sql on multiple tables. I then render the result set. For one table I would do:
sql = "select * from my_table"
results = my_table.objects.raw(sql)
For multiple tables I am doing:
sql = "select * from my_table, my_other_table where ...."
results = big_model.objects.raw(sql)
But, do I really need to create a table/model/class big_model, which contains all fields that I may need? I will never actually store any data in this "table".
ADDED:
I have a table my_users. I have a table my_listings. These are defined in Models.py. The table my_listings has a foreign key to my_users, indicating who created the listing.
The SQL is
"select user_name, listing_text from my_listings, my_users where my_users.id = my_listings.my_user_id".
I want this SQL to generate a result set that I can use to render my page in django.
The question is: Do I have to create a model that contains the fields user_name and listing_text? Or is there some better way that still uses raw SQL (select, from, where)? Of course, my actual queries are more complicated than this example. (The models that I define in models.py become actual tables in the database hence the use of the model/table term. Not sure how else to refer to them, sorry.) I use raw sql because I found that python table references only work with simple data models.

This works. Don't know why it didn't before :( From Dennis Baker's comment:
You do NOT need to have a model with all the fields in it, you just need the first model and fields from that. You do need to have the fields with unique names and as far as I know you should use "tablename.field as fieldname" to make sure you have all unique fields. I've done some fairly complex queries with 5+ tables this way and always tie them back to a single model. –
2 . Another solution is to use a cursor. However, a cursor has to be changed from a list of tuples to a list of dictionaries. I'm sure there are cleaner ways using iterators, but this function works. It takes a string, which is the raw sql query, and returns a list which can be rendered and used in a template.
from django.db import connection, transaction
def sql_select(sql):
cursor = connection.cursor()
cursor.execute(sql)
results = cursor.fetchall()
list = []
i = 0
for row in results:
dict = {}
field = 0
while True:
try:
dict[cursor.description[field][0]] = str(results[i][field])
field = field +1
except IndexError as e:
break
i = i + 1
list.append(dict)
return list

you do not need a model that includes the fields that you want to return from your raw sql. If you happen to have a model that actually has the fields that you want to return from your raw sql then you can map your raw sql output to this model, otherwise you can use cursors to go around models altogether.