We Keep Coding

update with csv file using python

I have to update the database with the CSV files. Consider the database table looks like this:
The CSV file data looks like this:
As you can see the CSV file data some data modified and some new records are added and what I supposed to do is to update only the data which is modified or some new records which are added.
In Table2 the first record of col2 is modified.. I need to update only the first record of col2(i.e, AA) but not the whole records of col2.
I could do this by hardcoding but I don't want to do it by hardcoding as I need to do this with 2000 tables.
Can anyone suggest me the steps to approach my goal.
Here is my code snippet..
df = pd.read_csv('F:\\filename.csv', sep=",", header=0, dtype=str)
sql_query2 = engine.execute('''
SELECT
*
FROM ttcmcs023111temp
''')
df2 = pd.DataFrame(sql_query2)
df.update(df2)

Since I do not have data similar to you, I used my own DB.
The schema of my books table is as follows:
+--------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------+-------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| name | varchar(30) | NO | | NULL | |
| author | char(30) | NO | | NULL | |
+--------+-------------+------+-----+---------+-------+
And the table looks like this:
+----+--------------------+------------------+
| id | name | author |
+----+--------------------+------------------+
| 1 | Origin | Dan Brown |
| 2 | River God | Wilbur Smith |
| 3 | Chromosome 6 | Robin Cook |
| 4 | Where Eagles Dare | Alistair Maclean |
| 5 | The Seventh Scroll | Dan Brown | ### Added wrong entry to prove
+----+--------------------+------------------+ ### my point
So, my approach is to create a new temporary table with the same schema as the books table from the CSV using python.
The code I used is as follows:
sql_query = sqlalchemy.text("CREATE TABLE temp (id int primary key, name varchar(30) not null, author varchar(30) not null)")
result = db_connection.execute(sql_query)
csv_df.to_sql('temp', con = db_connection, index = False, if_exists = 'append')
Which creates a table like this:
+----+--------------------+------------------+
| id | name | author |
+----+--------------------+------------------+
| 1 | Origin | Dan Brown |
| 2 | River God | Wilbur Smith |
| 3 | Chromosome 6 | Robin Cook |
| 4 | Where Eagles Dare | Alistair Maclean |
| 5 | The Seventh Scroll | Wilbur Smith |
+----+--------------------+------------------+
Now, you just need to use the update in MySQL using INNER JOIN to update the values you want to update in your original table. (in my case, 'books').
Here's how you'll do this:
statement = '''update books b
inner join temp t
on t.id = b.id
set b.name = t.name,
b.author = t.author;
'''
db_connection.execute(statement)
This query will update the values in table books from the table temp that I've created using the CSV.
You can destroy the temp table after updating the values.

Reference a Many-To-Many row

I am dealing with the design of a database in Flask connected to Postgresql. I have 2 Tables Reservation and Device which are related through a many-to-many relationship Table ReservationItem as follows:
| Reservation | | Device | | ReservationItem |
| ----------- | | ------ | | --------------- |
| id_res | | id_dev | | res_id (FK/PK) |
| etc... | | etc.. | | dev_id (FK/PK) |
| created_at |
| status |
Where dev_id and res_id are foreign keys and make up the composite primary key for the table. The columns created_at and status where originally conceived to track the history of the development of each Reservation-Device status.
Example
Someone reserves 3 Devices (respectively with id_dev's 1 - 2 - 3) on the 1st of January 2021 hence I would create 1 Reservation entry (id_res 1) and 3 ReservationItem entry with status "booked".
ReservationItem
| --------------------------------------|
| res_id | dev_id | created_at | status |
| ------------------------------------- |
| 1 | 1 | 2021-01-01 | booked |
| 1 | 2 | 2021-01-01 | booked |
| 1 | 3 | 2021-01-01 | booked |
On the 2nd of January the client returns the Device.id = 1 so I would create a fourth entry in the ReservationItem Table where the only updated fields are created_at and status, so that I could track where the devices are.
| --------------------------------------- |
| res_id | dev_id | created_at | status |
| --------------------------------------- |
| 1 | 1 | 2021-01-01 | booked |
| ... | ... | ... | ... |
| 1 | 1 | 2021-01-02 | returned |
Which basically weaken the uniqueness of the composite key (res_id,dev_id).
So I thought: Should I created another table lets say History to track these updates?
These would be my new models...
| ReservationItem | | History |
| --------------- | | ------------- |
| id_assoc (PK) | | id_hist (PK) |
| res_id (FK) | | assoc_id (FK) |
| dev_id (FK) | | created_at |
| | | status |
I would change the ReservationItem Table so that res_id are dev_id are not primary keys anymore. I would move the created_at and status into the History table and I would add the column id_assoc and use it as primary key, so that I can reference it from the History table.
I've been looking around and it seems that using one column as primary key in a many to many relationship is not ideal.
How would you design the relationships otherwise?
Is there any tool that Flask offers?
EDIT
After reading this post, which suggests to audit database table and write logs to track changed entries (or operations on databases) I found this article which suggests how to implement audit logs in Flask. But why wouldn't my solution work (or lets say "isn't ideal")?
thank you!

Sqlalchemy many to one array response

Im working with SQLAlchemy and Flask. I have a content table like:
--------------------------------------------
| id | title | description |
--------------------------------------------
| 1 | example | my content |
| 2 | another piece| my other content|
--------------------------------------------
And a status table like this:
--------------------------------------------------------
| id | content_id | status type | date |
--------------------------------------------------------
| 1 | 1 | written | 1/5/2020 |
| 2 | 1 | edited | 1/7/2020 |
--------------------------------------------------------
I want to be able to query the db and get a content with all of the status's in one row instead of have multiple rows of the content repeated. For example I want:
----------------------------------------------------------
| id | title | description | status's |
----------------------------------------------------------
| 1 | example | my content | [1,2] |
----------------------------------------------------------
Is there a way to do this with sqlalchemy?

You can use this query for fetching your answer:
SELECT b.*,
(SELECT GROUP_CONCAT (id) FROM status_table
WHERE content_id = b.id) AS `status's`
FROM status_table a JOIN content_table b
ON a.content_id = b.id
GROUP BY a.content_id;

Extracting Queryset of latest records based on filtered columns in Django

EDIT
I searched solution for my problem from stackoverflow and django documentation, but no luck. My models:
class User(models.Model):
name = models.CharField(max_length=10)
type = models.CharField(max_length=10)
class Userlog(models.Model):
user = models.ForeignKey('User', related_name='userlog')
pc_type = models.CharField(max_length=10)
login_date = models.DateField()
This table below is very simplified version of my sqlite table in Django app:
Table User:
+----+------+---------+
| ID | User | Type |
+----+------+---------+
| 1 | A | Admin |
| 2 | B | Admin |
| 3 | C | User |
+----+------+---------+
Table Userlog:
+----+---------+------------+------------+
| ID | user | pc_type | login_date |
+----+---------+------------+------------+
| 1 | A | Desktop | 2017/01/01 |
| 2 | A | Server | 2017/01/05 |
| 3 | B | Desktop | 2017/01/11 |
| 4 | A | Server | 2017/02/03 |
| 5 | C | Desktop | 2017/02/09 |
| 6 | B | Server | 2017/02/21 |
| 7 | A | Desktop | 2017/03/18 |
| 8 | C | Desktop | 2017/03/31 |
+----+---------+------------+------------+
I tried different approaches like that:
q = Userlog.objects.values('login_date').annotate(last_date =\
Max('login_date')).filter(pc_type='Desktop', user='Admin', login_date=F('last_date'))
But cannot extract latest dates for filtered columns.
I need Django QuerySet expression to get the result below:
(Latest login dates of Admins logged using Desktop)
+----+---------+------------+------------+
| ID | User_ID | pc_type | login_date |
+----+---------+------------+------------+
| 3 | B | Desktop | 2017/01/11 |
| 7 | A | Desktop | 2017/03/18 |
+----+---------+------------+------------+
I found similar question, but I want to use django expressions.

In Django documentation you can read more about annotate.
But in your case, try:
User.objects\
.filter(type='Admin', user_log__pc_type='Desktop')\
.annotate(last_login_date=Max('user_log__login_date'))
With .filter - you filter records you need
With .annotate - you get Maximum login date
To run exactly this code, you need to have related_name='user_log' for relation between User and UserLog:
class UserLog(models.Model)
user = models.ForeignKey(User, related_name='user_log')
...

Try :
Userlog.objects\
.filter(pc_type='Desktop', user__type='Admin')\
.order_by('-login_date')\
.disctinct('user')
You can access to user throught userlog.user.
Please, paste your models for exact query.

flask-sqlalchemy count function

Consider a table named result with the following schema
+----+-----+---------+
| id | tag | user_id |
+----+-----+---------+
| 0 | A | 0 |
| 1 | A | 0 |
| 2 | B | 0 |
| 3 | B | 0 |
+----+-----+---------+
for user with id=0 I would like to count they number of times a result with tag=A has been appeared. For now I have implemented it using raw SQL statement
db.session.execute('select tag, count(tag) from result where user_id = :id group by tag', {'id':user.id})
How can I write it using flask-sqlalchemy APIs?
Most of results I get mention the sqlalchemy function db.func.count() which is not available in flask-sqlalchemy or has a different path which I am not aware of.

I was using PyCharm as my IDE and it was not showing module members correctly, hence I thought count is missing. Here is my solution for the above
user.results.add_columns(Result.tag, db.func.count(Result.tag)).group_by(Result.tag).all()