We Keep Coding

mysql.connector do not give the last database state in Python

I use mysql.connector library in Python to send query to database. But, when the database is changed after the initialization, the mysql.connector’s tools answer like if the database had never change.
As example, let’s imagine I have a minimalist table students with just two columns id and name.
+----+------+
| id | name |
+----+------+
| 0 | foo |
+----+------+
In the following code, the query will ask the user with id 0. But, inside the process, some events will happened from outside the Python script and alter the database.
import mysql.connector
maindb = mysql.connector.connect(
host = "<host>",
user = "<user>",
password = "<password>",
db = "<database name>"
)
cursor = maindb.cursor()
# Here, I will send outside the python script a MySQL query to modify the name of the student from “foo” to “bar” like this:
# `UPDATE `students` SET `name` = 'bar' WHERE `students`.`id` = 0;`
cursor.execute("SELECT `id`, `name` FROM `students` WHERE `id` = 0")
result = cursor.fetchall()
print(result)
Then I get this answer [(0, 'foo')]. As you see, Python is not aware the data base has change since maindb.cursor() was called. So I get foo as name field instead of bar as expected.
So how to tell mysql.connector’s tools to take the last updates from the database when I send a query?

You will need to use a socket or if the changes occur frequently have your code re-run every x minutes

I just need to .connect() maindb object and .close() it before each new need.
maindb.connect()
cursor.execute("SELECT `id`, `name` FROM `students` WHERE `id` = 0")
result = cursor.fetchall()
print(result)
maindb.close()

The database maintains data integrity by preventing in-progress transactions from seeing changes made by other transactions (see transaction isolation levels).
You can commit your connection to allow it to see new changes:
cursor = maindb.cursor()
# Here, I will send outside the python script a MySQL query to modify the name of the student from “foo” to “bar” like this:
# `UPDATE `students` SET `name` = 'bar' WHERE `students`.`id` = 0;`
# Doesn't show the update
cursor.execute("SELECT `id`, `name` FROM `students` WHERE `id` = 0")
result = cursor.fetchall()
print(result)
# Shows the update because we have committed.
maindb.commit()
cursor.execute("SELECT `id`, `name` FROM `students` WHERE `id` = 0")
result = cursor.fetchall()
print(result)

Execute SQL for different where clause from Python

I am trying to print SQL result through python code, where I an trying to pass different predicates of the where clause from a for loop. But the code only taking the last value from the loop and giving the result.
In the below example I have two distinct id values 'aaa' and 'bbb'. There are 4 records for id value = 'aaa' and 2 records for the id value = 'bbb'.
But the below code only giving me the result for the id value ='bbb' not for id value 'aaa'
Can anyone help to identify what exactly wrong I am doing?
import pymysql
db = pymysql.connect(host="localhost", user="user1", passwd="pass1", db="db1")
cur = db.cursor()
in_lst=['aaa', 'bbb']
for i in in_lst:
Sql = "SELECT id, val, typ FROM test123 Where id='{inpt}'".format(inpt=i)
print(Sql)
cur.execute(Sql)
records = cur.fetchall()
print(records)
db.close()
The result I am getting as below
C:\Python34\python.exe C:/Users/Koushik/PycharmProjects/Test20161204/20170405.py
SELECT id, val, typ FROM test123 Where id='bbb'
(('bbb', 5, '1a'), ('bbb', 17, '1d'))
Process finished with exit code 0

import pymysql
db = pymysql.connect(host="localhost", user="root", passwd="1234", db="sakila")
cur = db.cursor()
in_lst=['1', '2']
for i in in_lst:
Sql = "SELECT * FROM actor Where actor_id='{inpt}'".format(inpt=i)
print(Sql)
cur.execute(Sql)
records = cur.fetchall()
print(records)
db.close()
Indentation is your problem, please update the code according to your needs...

Within your for loop, you're formatting the sql statement to replace "{inpt}" with "aaa". However, before you do anything with that value, you're immediately overwriting it with the "bbb" version.
You would need to either:
Store the results somehow before the next iteration of the loop, then process them outside of the loop.
Process the results within the loop.
Something like the following will give you a list containing both results from the fetchall() calls:
import pymysql
db = pymysql.connect(host="localhost", user="user1", passwd="pass1", db="db1")
cur = db.cursor()
in_lst=['aaa', 'bbb']
records = list()
for i in in_lst:
Sql = "SELECT id, val, typ FROM test123 Where id='{inpt}'".format(inpt=i)
print(Sql)
cur.execute(Sql)
records.append(cur.fetchall())
print(records)
db.close()

Python not able to modify MySQL but user can

Sorry if this question is stupid, I am 2 days into learning python
I have been beating my head against a wall trying to understand why my python script can run SELECT statements but not UPDATE or DELETE statements.
I believe this would be a MySQL issue and not a Python issue but I am no longer able to troubleshoot
pcheck.py
import re
import time
import json
import MySQLdb
import requests
from array import *
conn = MySQLdb.connect([redacted])
cur = conn.cursor()
sql1 = "SELECT pkey,pmeta FROM table1 WHERE proced = 0 LIMIT 1"
cur.execute(sql1)
row = cur.fetchone()
while row is not None:
print "row is: ",row[0]
rchk = [
r"(SHA256|MD5)",
r"(abc|def)"
]
for trigger in rchk:
regexp = re.compile(trigger)
pval = row[1]
if regexp.search(pval) is not None:
print "matched on: ",row[0]
sql2 = """INSERT INTO table2 (drule,dval,dmeta) VALUES('%s', '%s', '%s')"""
try:
args2 = (trigger, pval, row[1])
cur.execute(sql2, args2)
print(cur._last_executed)
except UnicodeError:
print "pass-uni"
break
else:
pass
sql3 = """UPDATE table1 SET proced=1 WHERE pkey=%s"""
args3 = row[0]
cur.execute(sql3, args3)
print(cur._last_executed)
row = cur.fetchone()
sql3 = """DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)"""
cur.execute(sql3)
print(cur._last_executed)
cur.close()
conn.close()
print "Finished"
And the actual (and suprisingly expected) output:
OUTPUT
scrape#:~/python$ python pcheck.py
row is: 0GqQ0d6B
UPDATE table1 SET proced=1 WHERE pkey='0GqQ0d6B'
DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)
Finished
However, the database is not being UPDATED. I checked that the query was making it to MySQL:
MySQL Log
"2015-12-14 22:53:56","localhost []","110","0","Query","SELECT `pkey`,`pmeta` FROM `table1` WHERE `proced`=0 LIMIT 200"
"2015-12-14 22:53:57","localhost []","110","0","Query","UPDATE `table1` SET `proced`=1 WHERE `pkey`='0GqQ0d6B'"
"2015-12-14 22:53:57","localhost []","110","0","Query","DELETE FROM table1 WHERE proced=1 AND last_update < (NOW() - INTERVAL 6 MINUTE)"
However proced value for row 0GqQ0d6B is still NOT 1
If I make the same queries via Sqlyog (logged in as user) the queries work as expected.

These kind of issues can be very frustrating. You sure there's no extra spaces here?
print "row is:*"+row[0]+"*"
Perhaps comment out the
for trigger in rchk:
section, and sprinkle some print statements around?

As the commenter Bob Dylan was able to deduce the cursor needed to be committed after the change.

python: psycopg2: psql: Delete row where condition

I'm trying to delete rows from a psql table on a condition.
I want all rows to be deleted if column "TagNaam" equals a variable var_tagnaam.
I've tried the following code and some variants but I can't get it to work.
There aren't any errors though.
cur.execute("DELETE FROM opc_actuelewaardentags WHERE 'TagNaam' = %s", (var_tagnaam,))
Is there something wrong with the syntax?
Edit:
Maybe it is more clear with additional code, the error might be in the other code?
for i in range(len(taginhoud)):
(var_tagnaam, var_tagwaarde, var_tagkwaliteit, var_tagtime) = taginhoud[i]
print (var_tagnaam)
cur.execute("DELETE FROM opc_actuelewaardentags WHERE 'TagNaam' = %s", (var_tagnaam,))
conn.commit()
cur.execute('INSERT INTO opc_actuelewaardentags ("TagNaam", "TagWaarde", "TagKwaliteit", create_date, write_date) VALUES (%s,%s,%s,now(),now())',
(var_tagnaam, var_tagwaarde, var_tagkwaliteit))
conn.commit()
So what I try to do here is:
Retrieve "var_tagnaam" from list "taginhoud".
Then in table opc_actuelewaardentags find all rows where column "Tagnaam" equals the value in "var_tagnaam". (Should be a string)
Then delete those rows where "Tagnaam" = "var_tagnaam". This part doesn't work.
Then insert new rows with data. This part works.
Could this code be wrong to do what I want?
I have tried many things already to solve the upper/lower case problem.
Edit 2:Query in pgadmin worked, trying to do the same thing in python:
I ran this query in pgadmin and it deleted the rows:
delete FROM opc_actuelewaardentags where "TagNaam" = 'Bakkerij.Device1.DB100INT8';
My attempt to make it as similar as possible in python:
var_tagnaam2 = "'"+var_tagnaam+"'"
cur.execute("DELETE FROM opc_actuelewaardentags WHERE \"TagNaam\" = %s", (var_tagnaam2,))
conn.commit()
Tried to escape the double quotes in attempt to make it the same as in pgadmin.

'TagNaam' is not a valid column_name identifier in sql language. You must not use single or double quotes in writing database name, table name or colunm name, but you can use apostrophe (`) .
Invalid:
DELETE FROM opc_actuelewaardentags WHERE 'TagNaam' = 'test';
DELETE FROM opc_actuelewaardentags WHERE "TagNaam" = 'test';
Valid:
DELETE FROM opc_actuelewaardentags WHERE TagNaam = 'test';
DELETE FROM opc_actuelewaardentags WHERE `TagNaam` = 'test';
DELETE FROM opc_actuelewaardentags WHERE "TagNaam" = 'test';
Update: According to PSQL dosc, double quote is a valid character in table and column names. It is especially used for key words while usinga as a table or column name. So following is valid:
DELETE FROM opc_actuelewaardentags WHERE "TagNaam" = 'test';
More is here...

I don't have a psql server, but a mysql server.
For MySQL:
mysql> select * from user where '1' = '1';
+------+
| id |
+------+
| 2 |
| 1 |
+------+
2 rows in set (0.05 sec)
mysql> select * from user;
+------+
| id |
+------+
| 2 |
| 1 |
+------+
2 rows in set (0.00 sec)
mysql> select * from user where '1' = "1";
+------+
| id |
+------+
| 2 |
| 1 |
+------+
2 rows in set (0.00 sec)
mysql> select * from user where 'id' = "1";
Empty set (0.00 sec)
mysql> select * from user where 'id' = 1;
Empty set, 1 warning (0.02 sec)
mysql> select * from user where id = 1;
+------+
| id |
+------+
| 1 |
+------+
1 row in set (0.02 sec)
mysql> select * from user where 'id' = "id";
+------+
| id |
+------+
| 2 |
| 1 |
+------+
2 rows in set (0.00 sec)
The SQL grammar should be similar. Therefore,
cur.execute("DELETE FROM opc_actuelewaardentags WHERE 'TagNaam' = %s", (var_tagnaam,))
should be
cur.execute("DELETE FROM opc_actuelewaardentags WHERE TagNaam = %s", (var_tagnaam,))
or
cur.execute("DELETE FROM opc_actuelewaardentags WHERE `TagNaam` = %s", (var_tagnaam,))
Above analyusis is error.
Simple Postgresql Statement - column name does not exists gives the answer.

RobbeM wrote: Edit 2:Query in pgadmin worked, trying to do the same thing in python
I've had the same symptoms - I could delete table rows using pgadmin or in SQL console, but Python code wouldn't work. The thing was I was accidentally creating cursor before establishing connection with postgreSQL server:
c = db_conn.cursor()
db_conn = psycopg2.connect(conn_string)
So, the solution for me was to create cursor after establishing connection with database:
db_conn = psycopg2.connect(conn_string)
c = db_conn.cursor()

Psycopg request hangs even if cursor is closed

We use an object that keeps connection to PostgreSQL database and creates new cursors to serve requests. I observed strange behavior: even when the response was read and the cursor is closed, the request is still hanging in the database, preventing updating the table etc etc.
When the connection is closed, it disappears.
I know about ORM frameworks and maybe will end up using one of them, but I just want to understand what's happening here. Why the request is still there?
Here's the python code:
import psycopg2
def main():
conn = psycopg2.connect("dbname=tmpdb password=1 host=localhost")
cur = conn.cursor()
cur.execute("SELECT 1;")
items = cur.fetchall()
cur.close()
#uncommenting the following line solves the problem
#conn.close()
print items
while True:
pass
main()
Here's how to start the code:
>python test_loop.py
[(1,)]
Here's how to observe hanging request:
tmpdb=# SELECT datname,usename,pid,client_addr,waiting,query_start,query FROM pg_stat_activity ;
datname | usename | pid | client_addr | waiting | query_start | query
---------+----------+-------+-------------+---------+-------------------------------+------------------------------------------------------------------------------------------
tmpdb | savenkov | 530 | ::1 | f | 2013-08-12 13:56:32.652996+00 | SELECT 1;
tmpdb | savenkov | 88351 | | f | 2013-08-12 13:56:35.331442+00 | SELECT datname,usename,pid,client_addr,waiting,query_start,query FROM pg_stat_activity ;
(2 rows)

Why do you think it is blocking?
Create the table
create table t (i integer);
Now run it:
import psycopg2
def main():
conn = psycopg2.connect("dbname=cpn")
cur = conn.cursor()
cur.execute("SELECT i from t;")
items = cur.fetchall()
print items
raw_input('Enter to insert')
cur.execute("insert into t (i) values (1) returning i;")
items = cur.fetchall()
conn.commit()
cur.execute("SELECT i from t;")
items = cur.fetchall()
print items
raw_input('Enter to update')
cur.execute("update t set i = 2 returning i")
items = cur.fetchall()
conn.commit()
cur.execute("SELECT i from t;")
items = cur.fetchall()
print items
cur.close()
while True:
pass
main()
Notice that you need to connection.commit() for it to be commited.
With that said don't do connection management. In instead use a connection pooler like Pgbouncer. It will save you from lots of complexity and frustration.
If the application runs on the same machine as the db then don't even bother. Just always close the connection as frequently as necessary. If both are in a fast intranet it is also not worth the added complexity of a connection pooler unless there is a really huge number of queries.