I have a simple application(Telegram bot) that about 2000-3000 people are using right now.
So I want to increase the balance of users every "N" seconds. It depends on their current status. The code works fine, but the function might not work after some amount of time. I started with 30 seconds, but after the first issue, I thought that 30 seconds is not enough to go through all rows and execute it. So right now I'm running it with 1200 seconds but anyway it stops growing after a while.
So is it just because of it or I'm doing something wrong in the code itself?
P.S. I'm using Python3 and SQLite and the bot is running constantly on a cheap, weak VPS server.
def balance_growth():
try:
cursor = connMembers.cursor()
sql = "SELECT * FROM members"
cursor.execute(sql)
data = cursor.fetchall()
for single_data in data:
if single_data[5] == "Basic":
sql = "UPDATE members SET balance = {B} + 1 WHERE chat_id = {I}".format(B=single_data[1], I=single_data[0])
cursor.execute(sql)
elif single_data[5] == "Bronze":
sql = "UPDATE members SET balance = {B} + 2 WHERE chat_id = {I}".format(B=single_data[1], I=single_data[0])
cursor.execute(sql)
elif single_data[5] == "Silver":
sql = "UPDATE members SET balance = {B} + 12 WHERE chat_id = {I}".format(B=single_data[1], I=single_data[0])
cursor.execute(sql)
elif single_data[5] == "Gold":
sql = "UPDATE members SET balance = {B} + 121 WHERE chat_id = {I}".format(B=single_data[1], I=single_data[0])
cursor.execute(sql)
elif single_data[5] == "Platinum":
sql = "UPDATE members SET balance = {B} + 1501 WHERE chat_id = {I}".format(B=single_data[1], I=single_data[0])
cursor.execute(sql)
cursor.execute(sql)
connMembers.commit()
cursor.close()
t = threading.Timer(120, balance_growth).start()
except Exception as err:
print(err)
Why not just do it all in a single update statement instead of one per row? Something like
UPDATE members SET balance = balance + (CASE whatever_column
WHEN "Platinum" THEN 1501
WHEN "Gold" THEN 121
WHEN "Silver" THEN 12
WHEN "Bronze" THEN 2
ELSE 1 END)
Edit:
Other suggestions:
Use integers instead of strings for the different levels, which will both be faster to compare and take up less space in the database.
Redesign your logic to not need an update every single tick. Maybe something like keeping track of the last time a row's balance was updated, and updating it according to the difference in time between then and now whenever you need to check the balance.
The problem is that you're calling commit() after every single UPDATE statement, which forces the database to write back all changes from its cache.
Do a single commit after you have finished everything.
Related
I currently have a list of id's approx. of size 10,000. I need to update all rows in the mySQL table which have an id in the inactive_ids list that you see below. I need to change their active status to 'No' which is a column in the mySQL table.
I am using mysql.connector python library.
When I run the code below, it is taking about 0.7 seconds to execute each iteration in the for loop. Thats about a 2 hour run time for all 10,000 id's to be changed. Is there a more optimal/quicker way to do this?
# inactive_ids are unique strings something like shown below
# inactive_ids = ['a9okeoko', 'sdfhreaa', 'xsdfasy', ..., 'asdfad']
# initialize connection
mydb = mysql.connector.connect(
user="REMOVED",
password="REMOVED",
host="REMOVED",
database="REMOVED"
)
# initialize cursor
mycursor = mydb.cursor(buffered=True)
# Function to execute multiple lines
def alter(state, msg, count):
result = mycursor.execute(state, multi=True)
result.send(None)
print(str(count), ': ', msg, result)
count += 1
return count
# Try to execute, throw exception if fails
try:
count = 0
for Id in inactive_ids:
# SAVE THE QUERY AS STRING
sql_update = "UPDATE test_table SET Active = 'No' WHERE NoticeId = '" + Id + "'"
# ALTER
count = alter(sql_update, "done", count)
# commits all changes to the database
mydb.commit()
except Exception as e:
mydb.rollback()
raise e
Do it with a single query that uses IN (...) instead of multiple queries.
placeholders = ','.join(['%s'] * len(inactive_ids))
sql_update = f"""
UPDATE test_table
SET Active = 'No'
WHERE NoticeId IN ({placeholders})
"""
mycursor.execute(sql_update, inactive_ids)
I've a function in my code which is as follows:
async def register():
db = connector.connect(host='localhost',user='root',password='root',database='testing')
cursor = db.cursor()
cursor.execute('LOCK TABLES Data WRITE;')
cursor.execute('SELECT Total_Reg FROM Data;')
data = cursor.fetchall()
reg = data[0][0]
print(reg)
if reg >= 30:
print("CLOSED!")
return
await asyncio.sleep(1)
cursor.execute('UPDATE Data SET Total_Reg = Total_Reg + 1 WHERE Id = 1')
cursor.execute('COMMIT;')
print("REGISTERED!")
db.close()
In case of multiple instances of this register function running at the same time, there is an unexpected infinite loop occurs blocking my entire code. Why is that so? Also, if it's a deadlock [I assume] then why my program is not raising any error? Please tell me why is this happening? And what can be done to prevent this issue?
A much simpler construct:
db = connector.connect(...
cursor = db.cursor()
if cursor.execute('UPDATE Data SET Total_Reg = Total_Reg + 1 WHERE Id = 1 AND Total_Reg < 30'):
print("REGISTERED!")
else:
print("CLOSED!")
db.close()
So:
Don't use LOCK TABLES, not until you understand transactions, and then rarely
Use the SQL to enforce the constraints you want
Use the return value to see if any rows where changed
Don't use sleep statements
My python script is stuck on cursor execution and I have problem to find out why.
db = cx_Oracle.connect('user/password#user')
cursor = db.cursor()
for i in range(1, 10):
insert_data = """
update products set prd_stock_holder = '~',prd_prod_quality = 'FREE'
where PRD_PRI_ID = (select prd_pri_id from products join PRODUCT_INFOS
on prd_pri_id = pri_id
where PRI_code = '%s')""" %i
cursor.execute(insert_data) # stuck here
print "product %s updated" %i # never printed
db.commit()
I had a simular issue with insert to a MySQL Database.
Moving the db.commit() inside the loop solved my issue.
db = cx_Oracle.connect('user/password#user')
cursor = db.cursor()
for i in range(1, 10):
insert_data = """
update products set prd_stock_holder = '~',prd_prod_quality = 'FREE'
where PRD_PRI_ID = (select prd_pri_id from products join PRODUCT_INFOS
on prd_pri_id = pri_id
where PRI_code = '%s')""" %i
cursor.execute(insert_data) # stuck here
db.commit()
print "product %s updated" %i # never printed
A cause may be a lock on the Oracle RDBMS. You may try checking if your session is locked waiting for a resource (e.g. http://www.dba-oracle.com/t_tracking_oracle_blocking_sessions.htm).
Please keep in mind that locks may be connected to the execution plan the db has decided to apply. By rewriting your update query you might benefit from thinner lock acquisition.
I am creating a program that will allow a user to pick 2 chemical elements from menus and then tell them the result of the reaction between those two. I know my database is working, and I am trying to create two variables that can be changed at any point in the program. This is the code I have so far, using 2 values that I know the correct outcome for:
import sqlite3
conn = sqlite3.connect('Experiment_simulator_database.db')
c = conn.cursor()
firstchoice = 1
secondchoice = 36
sqlcommand = "SELECT Outcome_ID FROM Reactions WHERE Choice_1 = firstchoice AND Choice_2 = secondchoice"
c.execute(sqlcommand)
result = c.fetchone()
print(result)
How can I get firstchoice and secondchoice in the select statement to take on the values I specified above?
You can have placeholders in your sql and bind the values when you call execute
sqlcommand = "SELECT Outcome_ID FROM Reactions WHERE Choice_1 = ? AND Choice_2 = ?"
c.execute(sqlcommand, (firstchoice, secondchoice,))
You can pass these variables in parameter where you are using.
firstchoice = 1
secondchoice = 36
sqlcommand = "SELECT Outcome_ID FROM Reactions WHERE Choice_1 = " + firstchoice + "AND Choice_2 = " + secondchoice "
c.execute(sqlcommand, firstchoice, secondchoice)
Sorry if this question is stupid, I am 2 days into learning python
I have been beating my head against a wall trying to understand why my python script can run SELECT statements but not UPDATE or DELETE statements.
I believe this would be a MySQL issue and not a Python issue but I am no longer able to troubleshoot
pcheck.py
import re
import time
import json
import MySQLdb
import requests
from array import *
conn = MySQLdb.connect([redacted])
cur = conn.cursor()
sql1 = "SELECT pkey,pmeta FROM table1 WHERE proced = 0 LIMIT 1"
cur.execute(sql1)
row = cur.fetchone()
while row is not None:
print "row is: ",row[0]
rchk = [
r"(SHA256|MD5)",
r"(abc|def)"
]
for trigger in rchk:
regexp = re.compile(trigger)
pval = row[1]
if regexp.search(pval) is not None:
print "matched on: ",row[0]
sql2 = """INSERT INTO table2 (drule,dval,dmeta) VALUES('%s', '%s', '%s')"""
try:
args2 = (trigger, pval, row[1])
cur.execute(sql2, args2)
print(cur._last_executed)
except UnicodeError:
print "pass-uni"
break
else:
pass
sql3 = """UPDATE table1 SET proced=1 WHERE pkey=%s"""
args3 = row[0]
cur.execute(sql3, args3)
print(cur._last_executed)
row = cur.fetchone()
sql3 = """DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)"""
cur.execute(sql3)
print(cur._last_executed)
cur.close()
conn.close()
print "Finished"
And the actual (and suprisingly expected) output:
OUTPUT
scrape#:~/python$ python pcheck.py
row is: 0GqQ0d6B
UPDATE table1 SET proced=1 WHERE pkey='0GqQ0d6B'
DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)
Finished
However, the database is not being UPDATED. I checked that the query was making it to MySQL:
MySQL Log
"2015-12-14 22:53:56","localhost []","110","0","Query","SELECT `pkey`,`pmeta` FROM `table1` WHERE `proced`=0 LIMIT 200"
"2015-12-14 22:53:57","localhost []","110","0","Query","UPDATE `table1` SET `proced`=1 WHERE `pkey`='0GqQ0d6B'"
"2015-12-14 22:53:57","localhost []","110","0","Query","DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)"
However proced value for row 0GqQ0d6B is still NOT 1
If I make the same queries via Sqlyog (logged in as user) the queries work as expected.
These kind of issues can be very frustrating. You sure there's no extra spaces here?
print "row is:*"+row[0]+"*"
Perhaps comment out the
for trigger in rchk:
section, and sprinkle some print statements around?
As the commenter Bob Dylan was able to deduce the cursor needed to be committed after the change.