We use an object that keeps connection to PostgreSQL database and creates new cursors to serve requests. I observed strange behavior: even when the response was read and the cursor is closed, the request is still hanging in the database, preventing updating the table etc etc.
When the connection is closed, it disappears.
I know about ORM frameworks and maybe will end up using one of them, but I just want to understand what's happening here. Why the request is still there?
Here's the python code:
import psycopg2
def main():
conn = psycopg2.connect("dbname=tmpdb password=1 host=localhost")
cur = conn.cursor()
cur.execute("SELECT 1;")
items = cur.fetchall()
cur.close()
#uncommenting the following line solves the problem
#conn.close()
print items
while True:
pass
main()
Here's how to start the code:
>python test_loop.py
[(1,)]
Here's how to observe hanging request:
tmpdb=# SELECT datname,usename,pid,client_addr,waiting,query_start,query FROM pg_stat_activity ;
datname | usename | pid | client_addr | waiting | query_start | query
---------+----------+-------+-------------+---------+-------------------------------+------------------------------------------------------------------------------------------
tmpdb | savenkov | 530 | ::1 | f | 2013-08-12 13:56:32.652996+00 | SELECT 1;
tmpdb | savenkov | 88351 | | f | 2013-08-12 13:56:35.331442+00 | SELECT datname,usename,pid,client_addr,waiting,query_start,query FROM pg_stat_activity ;
(2 rows)
Why do you think it is blocking?
Create the table
create table t (i integer);
Now run it:
import psycopg2
def main():
conn = psycopg2.connect("dbname=cpn")
cur = conn.cursor()
cur.execute("SELECT i from t;")
items = cur.fetchall()
print items
raw_input('Enter to insert')
cur.execute("insert into t (i) values (1) returning i;")
items = cur.fetchall()
conn.commit()
cur.execute("SELECT i from t;")
items = cur.fetchall()
print items
raw_input('Enter to update')
cur.execute("update t set i = 2 returning i")
items = cur.fetchall()
conn.commit()
cur.execute("SELECT i from t;")
items = cur.fetchall()
print items
cur.close()
while True:
pass
main()
Notice that you need to connection.commit() for it to be commited.
With that said don't do connection management. In instead use a connection pooler like Pgbouncer. It will save you from lots of complexity and frustration.
If the application runs on the same machine as the db then don't even bother. Just always close the connection as frequently as necessary. If both are in a fast intranet it is also not worth the added complexity of a connection pooler unless there is a really huge number of queries.
Related
I use mysql.connector library in Python to send query to database. But, when the database is changed after the initialization, the mysql.connector’s tools answer like if the database had never change.
As example, let’s imagine I have a minimalist table students with just two columns id and name.
+----+------+
| id | name |
+----+------+
| 0 | foo |
+----+------+
In the following code, the query will ask the user with id 0. But, inside the process, some events will happened from outside the Python script and alter the database.
import mysql.connector
maindb = mysql.connector.connect(
host = "<host>",
user = "<user>",
password = "<password>",
db = "<database name>"
)
cursor = maindb.cursor()
# Here, I will send outside the python script a MySQL query to modify the name of the student from “foo” to “bar” like this:
# `UPDATE `students` SET `name` = 'bar' WHERE `students`.`id` = 0;`
cursor.execute("SELECT `id`, `name` FROM `students` WHERE `id` = 0")
result = cursor.fetchall()
print(result)
Then I get this answer [(0, 'foo')]. As you see, Python is not aware the data base has change since maindb.cursor() was called. So I get foo as name field instead of bar as expected.
So how to tell mysql.connector’s tools to take the last updates from the database when I send a query?
You will need to use a socket or if the changes occur frequently have your code re-run every x minutes
I just need to .connect() maindb object and .close() it before each new need.
maindb.connect()
cursor.execute("SELECT `id`, `name` FROM `students` WHERE `id` = 0")
result = cursor.fetchall()
print(result)
maindb.close()
The database maintains data integrity by preventing in-progress transactions from seeing changes made by other transactions (see transaction isolation levels).
You can commit your connection to allow it to see new changes:
cursor = maindb.cursor()
# Here, I will send outside the python script a MySQL query to modify the name of the student from “foo” to “bar” like this:
# `UPDATE `students` SET `name` = 'bar' WHERE `students`.`id` = 0;`
# Doesn't show the update
cursor.execute("SELECT `id`, `name` FROM `students` WHERE `id` = 0")
result = cursor.fetchall()
print(result)
# Shows the update because we have committed.
maindb.commit()
cursor.execute("SELECT `id`, `name` FROM `students` WHERE `id` = 0")
result = cursor.fetchall()
print(result)
I have the following Postgres query where I am fetching data from table1 with rows ~25 million and would like to write the output of the below query into multiple files.
query = """ WITH sequence AS (
SELECT
a,
b,
c
FROM table1 )
select * from sequence;"""
Below is the python script to fetch the complete dataset. How can I modify the script to fetch it to multiple files (eg. each file has 10000 rows)
#IMPORT LIBRARIES ########################
import psycopg2
from pandas import DataFrame
#CREATE DATABASE CONNECTION ########################
connect_str = "dbname='x' user='x' host='x' " "password='x' port = x"
conn = psycopg2.connect(connect_str)
cur = conn.cursor()
conn.autocommit = True
cur.execute(query)
df = DataFrame(cur.fetchall())
Thanks
Here are 3 methods that may help
use psycopg2 named cursor cursor.itersize = 2000
snippet
with conn.cursor(name='fetch_large_result') as cursor:
cursor.itersize = 20000
query = "SELECT * FROM ..."
cursor.execute(query)
for row in cursor:
....
use psycopg2 named cursor fetchmany(size=2000)
snippet
conn = psycopg2.connect(conn_url)
cursor = conn.cursor(name='fetch_large_result')
cursor.execute('SELECT * FROM <large_table>')
while True:
# consume result over a series of iterations
# with each iteration fetching 2000 records
records = cursor.fetchmany(size=2000)
if not records:
break
for r in records:
....
cursor.close() # cleanup
conn.close()
Finally you could define the a SCROLL CURSOR
Define a SCROLL CURSOR
snippet
BEGIN MY_WORK;
-- Set up a cursor:
DECLARE scroll_cursor_bd SCROLL CURSOR FOR SELECT * FROM My_Table;
-- Fetch the first 5 rows in the cursor scroll_cursor_bd:
FETCH FORWARD 5 FROM scroll_cursor_bd;
CLOSE scroll_cursor_bd;
COMMIT MY_WORK;
Please note Not naming the cursor in psycopg2 will cause the cursor to be client side as opposed to server side.
I'm trying to optimize the below query and python script.
There are 100,000+ rows in file and about 300,000 items in file. It takes a very long time to run. Any ideas?
import MySQLdb.cursors as cursors
db = MySQLdb.connect(host=*,user=*,passwd=*,db=*, cursorclass=cursors.SSCursor)
cursor = db.cursor()
print 'running large query'
cursor.execute('
SELECT `file`.`file_id`,`file`.`format` FROM `file`
LEFT OUTER JOIN (SELECT `file_id`, `filename` FROM `file_version` WHERE `filename` = %s) AS `version`
ON `file`.`file_id` = `version`.`file_id` WHERE `version`.`filename` IS NULL',[thumbnail])
results = cursor.fetchall()
cursor.close()
for row in results
table file
file_id (pk) | format | etc.
table file_version
version_id (pk) | file_id | filename | etc.
Each file can have multiple versions, e.g. original.jpg, 300.jpg, 600.jpg, 800.jpg. The select inside a select is what is really slowing it down IMO. Thank you.
I am trying to do a insert query in the SQL. It indicates that it succeed but shows no record in the database. Here's my code
conn = MySQLdb.connect("localhost",self.user,"",self.db)
cursor = conn.cursor()
id_val = 123456;
path_val = "/homes/error.path"
host_val = "123.23.45.64"
time_val = 7
cursor.execute("INSERT INTO success (id,path,hostname,time_elapsed) VALUES (%s,%s,%s,%s)", (id_val, path_val,host_val,time_val))
print "query executed"
rows = cursor.fetchall()
print rows
this outputs the following
query executed
()
it gives me no errors but the database seems to be empty. I tried my SQL query in the mysql console. executed the following command.
INSERT INTO success (id,path,hostname,time_elapsed)
VALUES (1,'sometext','hosttext',4);
This works fine as I can see the database got populated.
mysql> SELECT * FROM success LIMIT 5;
+----+----------+----------+--------------+
| id | path | hostname | time_elapsed |
+----+----------+----------+--------------+
| 1 | sometext | hosttext | 4 |
+----+----------+----------+--------------+
so I am guessing the SQL query command is right. Not sure why my cursor.execute is not responding. Could someone please point me to the right direction. Can't seem to figure out the bug. thanks
After you are sending your INSERT record, you should commit your changes in the database:
cursor.execute("INSERT INTO success (id,path,hostname,time_elapsed) VALUES (%s,%s,%s,%s)", (id_val, path_val,host_val,time_val))
conn.commit()
When you want to read the data, you should first send your query as you did through your interpreter.
So before you fetch the data, execute the SELECT command:
cursor.execute("SELECT * FROM success")
rows = cursor.fetchall()
print rows
If you want to do it pythonic:
cursor.execute("SELECT * FROM success")
for row in cursor:
print(row)
[enter image description here][1]I am currently making a project for school and the time for reaching the wall has come.
I am trying to fetch data from the USB port on Raspberry Pi 3+. I have connected an Arduino Nano and I am sending a string(UID number in decimal of an RFID Card) from it to the Pi via the USB port. Everything works fine here, I can print out the string(ID) without a problem. I am comparing the ID from the card with the one in my database, and if I put a static number ( commented below in the code) it prints the data. However if I try with the serial line, nothing happens. It seems like it doesn't fetch the data at all. The outlook of my database is underneath and the python code as well.
Thanks in Advance !!
card_id serial_no LastName FirstName
1 | 2136106133 | Hansen | Peter |
2 | 117254270 | Larsen | Thompson |
#!/usr/bin/env python
import MySQLdb
import serial
ser = serial.Serial('/dev/ttyUSB0',9600)
db = MySQLdb.connect(host="localhost", # your host, usually localhost
user="root", # your username
passwd="root", # your password
db="RFID") # name of the data base
cur=db.cursor()
CardID = 0
LastName = ""
FirstName = ""
while True:
CardID=ser.readline()
print "pweasda"
print CardID
print "pewpew"
# CardID = 117254270 - this works. The problem is that I have many RFID cards/tags with different IDs of course. With this statement it prints everything correctly.
cur.execute('SELECT * FROM cards WHERE serial_no=%s',(CardID))
results = cur.fetchall()
for row in results:
FirstName = row[3]
LastName = row [2]
serial_no = row [1]
card_id = row [0]
#print the fetched results
print "FirstName=%s,LastName=%s,serial_no=%s,card_id=%s" % \
(FirstName, LastName, serial_no, card_id )
db.commit()
print "Data committed"
output image (no errors): [1]: http://postimg.org/image/jf2doogrv/
Possible solution could be:
import sqlite3
conn = sqlite3.connect("users.db")#path to your sqlite db file!
cursor = conn.cursor()
CardID=ser.readline().strip()
sql = "SELECT * FROM cards WHERE serial_no=?"
cursor.execute(sql, [(CardID)])
try:
results = cursor.fetchall()[0]# just fetching the first row only, you can loop through all rows here!
FirstName = results[3]
LastName = results[2]
serial_no = results[1]
card_id = results[0]
print "FirstName=%s,LastName=%s,serial_no=%s,card_id=%s" % \
(FirstName, LastName, serial_no, card_id )
except IndexError as e:
print 'CardID Not Exist'+str(e)
except Exception as e2:
print(str(e2))
In above code I am assuming the database is in sqlite DB, and also handled the exceptions so you can figure out the runtime error, if any!