How can one optimize this MySQL count algorithm? - python

I have 2 tables; one is users and the other records user actions. I want to count the number of actions per user and record this in the users table. There are ~100k user and the following code takes 6 hours! There must be a better way!
def calculate_invites():
sql_db.execute("SELECT id, uid FROM users")
for row in sql_db:
id = row['id']
uid = row['uid']
sql1 = "SELECT COUNT(1) FROM actions WHERE uid = %s"
sql_db.execute(sql1, uid)
count_actions = sql_db.fetchone()["COUNT(1)"]
sql = "UPDATE users SET count_actions=%s WHERE uid=%s"
sql_db.execute(sql, (count_actions, uid))

You can do this all as one statement:
update users
set count_actons = (select count(*) from actions a where a.uid = users.uid)
No for loop. No multiple queries. Do in SQL what you can do in SQL. Generally looping over rows is something you want to do in the database rather than in the application.

Offered only as an alternative since Gordon's answer is probably faster:
update users
from (
select uid, count(*) as num_actions
from actions
group by uid
) x
set count_actions = x.num_actions
where users.uid=x.uid

Related

how to run update sql query based on select query results using python

I am trying to update a row in an sql table based on results of a select query using python. How to add same so that if results found in select query we should be able to run update query and print as list updated, if not exit saying no result found to update table
cursor = conn.cursor()
cursor.execute(
"select * from student where email = 'xyz.com'"
)
student = cursor.fetchall()
print(student)
for row in student:
cursor.execute(" Update student set value = 0 where email = 'xyz.com'")
`
You don't need to do a separate SELECT, you can just use an OUTPUT clause to have your UPDATE statement return the value(s) from the row(s) that are updated. For example, with pyodbc:
sql = """\
UPDATE student SET value = 0
OUTPUT INSERTED.student_number
WHERE email = 'xyz.com' AND (value <> 0 OR value IS NULL)
"""
student_numbers = crsr.execute(sql).fetchall()
print(student_numbers) # [(1001, ), (1003, )]

Unable to fetch data using select query

I have used GCP cloud SQL with dailogflow CX there is one table QuestionPageMapping_test it stored all padeId and FlowID of questions that asked by bot. It is used for redo activity. If user wants to go back to any question then user can select that question according to that question select query will fetch pageID of that question and user can start their conversation from that question again. But select query does not work properly at some instance.It work for some questions.
option=r=re.sub(r'[^\w\s]', ' ', option)
connection2 = connector.connect(os.environ['DB_Instance_Name'],"pymysql",database='bot_info',user="root",password=os.environ['DB_Password'])
cursor2 = connection2.cursor(pymysql.cursors.DictCursor)
query1= f"select * from QuestionPageMapping_test where questions ='{option}'"
print(query1)
cursor2.execute(query1)
rows = cursor2.fetchall()
print(rows)
for row in rows:
global option2
option2=row["pageId"]
connection2.close()
query="set #row_number=0;"
cursor.execute(query)
query = f"select * FROM (select *,(#row_number:=#row_number + 1) AS R_NUM from {session_id} order by timestamp) as a1 where pageId='{option2}'"
print(query)
I tried this code,in that option variable is used for to store question that is present at dailoflow cx's page. when select query fail then rows is empty that is why it is not entering into for loop and because of that I am getting NameError: name 'option2' is not defined" for next query.
(https://i.stack.imgur.com/559BB.png)

Updating a record with a value from another table - MySQL (Python)

I am trying to update a value in one table (csms) in MySQL from another table (serviceppl). I have attached the query I wrote and both tables, but the query does not work. It does not produce any error, but the record doesn't get updated. Help!
query = "select*from csms where feedback in('null', 'I expect a better one', 'Bad one! Need one again')"
mycursor.execute(query)
p = mycursor.fetchall()
for i in p:
query_upserv = """update csms inner join serviceppl on csms.product = serviceppl.speciality
set serviceppl.name = csms.serviceman where csms.product = '{}'""".format(i[4])
mycursor.execute(query_upserv)
mycon.commit()
If you want to update the column serviceman of the table csms then you must fix your SET clause:
set csms.serviceman = serviceppl.name

Getting the id of the last record inserted for Postgresql SERIAL KEY with Python

I am using SQLAlchemy without the ORM, i.e. using hand-crafted SQL statements to directly interact with the backend database. I am using PG as my backend database (psycopg2 as DB driver) in this instance - I don't know if that affects the answer.
I have statements like this,for brevity, assume that conn is a valid connection to the database:
conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)")
Assume also that the user table consists of the columns (id [SERIAL PRIMARY KEY], name, country_id)
How may I obtain the id of the new user, ideally, without hitting the database again?
You might be able to use the RETURNING clause of the INSERT statement like this:
result = conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)
RETURNING *")
If you only want the resulting id:
result = conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)
RETURNING id")
[new_id] = result.fetchone()
User lastrowid
result = conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)")
result.lastrowid
Current SQLAlchemy documentation suggests
result.inserted_primary_key should work!
Python + SQLAlchemy
after commit, you get the primary_key column id (autoincremeted) updated in your object.
db.session.add(new_usr)
db.session.commit() #will insert the new_usr data into database AND retrieve id
idd = new_usr.usrID # usrID is the autoincremented primary_key column.
return jsonify(idd),201 #usrID = 12, correct id from table User in Database.
this question has been asked many times on stackoverflow and no answer I have seen is comprehensive. Googling 'sqlalchemy insert get id of new row' brings up a lot of them.
There are three levels to SQLAlchemy.
Top: the ORM.
Middle: Database abstraction (DBA) with Table classes etc.
Bottom: SQL using the text function.
To an OO programmer the ORM level looks natural, but to a database programmer it looks ugly and the ORM gets in the way. The DBA layer is an OK compromise. The SQL layer looks natural to database programmers and would look alien to an OO-only programmer.
Each level has it own syntax, similar but different enough to be frustrating. On top of this there is almost too much documentation online, very hard to find the answer.
I will describe how to get the inserted id AT THE SQL LAYER for the RDBMS I use.
Table: User(user_id integer primary autoincrement key, user_name string)
conn: Is a Connection obtained within SQLAlchemy to the DBMS you are using.
SQLite
======
insstmt = text(
'''INSERT INTO user (user_name)
VALUES (:usernm) ''' )
# Execute within a transaction (optional)
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = result.lastrowid
txn.commit()
MS SQL Server
=============
insstmt = text(
'''INSERT INTO user (user_name)
OUTPUT inserted.record_id
VALUES (:usernm) ''' )
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = result.fetchone()[0]
txn.commit()
MariaDB/MySQL
=============
insstmt = text(
'''INSERT INTO user (user_name)
VALUES (:usernm) ''' )
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = conn.execute(text('SELECT LAST_INSERT_ID()')).fetchone()[0]
txn.commit()
Postgres
========
insstmt = text(
'''INSERT INTO user (user_name)
VALUES (:usernm)
RETURNING user_id ''' )
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = result.fetchone()[0]
txn.commit()
result.inserted_primary_key
Worked for me. The only thing to note is that this returns a list that contains that last_insert_id.
Make sure you use fetchrow/fetch to receive the returning object
insert_stmt = user.insert().values(name="homer", country_id="123").returning(user.c.id)
row_id = await conn.fetchrow(insert_stmt)
For Postgress inserts from python code is simple to use "RETURNING" keyword with the "col_id" (name of the column which you want to get the last inserted row id) in insert statement at end
syntax -
from sqlalchemy import create_engine
conn_string = "postgresql://USERNAME:PSWD#HOSTNAME/DATABASE_NAME"
db = create_engine(conn_string)
conn = db.connect()
INSERT INTO emp_table (col_id, Name ,Age)
VALUES(3,'xyz',30) RETURNING col_id;
or
(if col_id column is auto increment)
insert_sql = (INSERT INTO emp_table (Name ,Age)
VALUES('xyz',30) RETURNING col_id;)
result = conn.execute(insert_sql)
[last_row_id] = result.fetchone()
print(last_row_id)
#output = 3
ex -

using results from a sql query in a python program in another sql query

sorry for my previous question which was very ambiguous, but i think if i get the answer to this question I can work it out.
In the program below i have selected the barcodes of products where the amount is less than the quantity. I want to say, that if the barcodes(in the fridge table) match barcodes in another table(products), set the stock field equal to 0. The problem Im getting is that the program is trying to match all the barcodes that it found in the query against single barcodes in the products table(thats what I think). does anyone know what to do. thanks a million. lincoln.
import MySQLdb
def order():
db = MySQLdb.connect(host='localhost', user='root', passwd='$$', db='fillmyfridge')
cursor = db.cursor()
cursor.execute('select barcode from fridge where amount < quantity')
db.commit()
row = cursor.fetchall()
cursor.execute('update products set stock = 0 where barcode = %s', row)
UPDATE products SET stock = 0 WHERE barcode IN (
SELECT fridge.barcode FROM fridge WHERE fridge.amount < fridge.quantity );
I know this doesn't answer the question exactly but two SQL statements are not required.
To do it in python:
import MySQLdb
def order():
db = MySQLdb.connect(host='localhost', user='root', passwd='$$', db='fillmyfridge')
cursor = db.cursor()
cursor.execute('select barcode from fridge where amount < quantity')
db.commit()
rows = cursor.fetchall()
for row in rows
cursor.execute('update products set stock = 0 where barcode = %s', row[0])
This is more of SQL query than Python, but still I will try to answer that:
(I haven't worked with MySQL but PostgreSQL, so there might slight variation in interpretation of things here).
when you did
cursor.execute('select barcode from fridge where amount < quantity')
db.commit()
row = cursor.fetchall()
the variable 'row' now is a resultset (to understand: a list of rows from the database)
something like [(barcode1), (barcode2), (barcode3)..]
when you do the update statement
cursor.execute('update products set stock = 0 where barcode = %s', row)
this turns into something like:
update products set stock = 0 where barcode = [(barcode1), (barcode2), (barcode3)..]
which is not a correct SQL statement.
you should do something like this:
cursor.execute('update products set stock = 0 where barcode in (%s)', ','.join([each[0] for each in row]))
or better, the optimized thing:
import MySQLdb
def order():
db = MySQLdb.connect(host='localhost', user='root', passwd='$$', db='fillmyfridge')
cursor = db.cursor()
cursor.execute('update products set stock = 0 where barcode in (select barcode from fridge where amount < quantity)')
db.commit()
Well, to add more you have a db.commit() after a select query and not after an update query, thats a basic fault. Select is idempotent, doesn't need commit, whereas Update does. I will strongly recommend you to go through some SQL before continuing.

Categories