I have some code like this
db = MySQLdb.connect(host="localhost", user="root", passwd="root", db="domains")
cursor = db.cursor()
def check_urls(res):
pool = Pool(25)
for row in res:
pool.spawn(fetch, row[0], row[1])
pool.join()
def fetch(*args):
"""code trimmed for brevity"""
cursor.execute("""UPDATE com SET http_status=%s, is_checked=1 WHERE id=%s""",
(output.get('http_status', ""), id))
for _ in xrange(10000):
cursor.execute("SELECT domain, id FROM com WHERE is_checked IS NULL LIMIT 100")
result = cursor.fetchall()
check_urls(result)
db.commit()
cursor.close()
db.close()
My program get stuck at db.commit(). No values updated in the database. Can someone tell me whats wrong?.
Please note:
My check_urls function has a for loop and each loop has one mysql update query.
It seems like you should move your db.commit() into fetch function after cursor.execute. Another way you may use "global" keyword with cursor. Or, third way, just make cursor as parameter of functions.
Related
I am inserting JSON data into a MySQL database
I am parsing the JSON and then inserting it into a MySQL db using the python connector
Through trial, I can see the error is associated with this piece of code
for steps in result['routes'][0]['legs'][0]['steps']:
query = ('SELECT leg_no FROM leg_data WHERE travel_mode = %s AND Orig_lat = %s AND Orig_lng = %s AND Dest_lat = %s AND Dest_lng = %s AND time_stamp = %s')
if steps['travel_mode'] == "pub_tran":
travel_mode = steps['travel_mode']
Orig_lat = steps['var_1']['dep']['lat']
Orig_lng = steps['var_1']['dep']['lng']
Dest_lat = steps['var_1']['arr']['lat']
Dest_lng = steps['var_1']['arr']['lng']
time_stamp = leg['_sent_time_stamp']
if steps['travel_mode'] =="a_pied":
query = ('SELECT leg_no FROM leg_data WHERE travel_mode = %s AND Orig_lat = %s AND Orig_lng = %s AND Dest_lat = %s AND Dest_lng = %s AND time_stamp = %s')
travel_mode = steps['travel_mode']
Orig_lat = steps['var_2']['lat']
Orig_lng = steps['var_2']['lng']
Dest_lat = steps['var_2']['lat']
Dest_lng = steps['var_2']['lng']
time_stamp = leg['_sent_time_stamp']
cursor.execute(query,(travel_mode, Orig_lat, Orig_lng, Dest_lat, Dest_lng, time_stamp))
leg_no = cursor.fetchone()[0]
print(leg_no)
I have inserted higher level details and am now searching the database to associate this lower level information with its parent. The only way to find this unique value is to search via the origin and destination coordinates with the time_stamp. I believe the logic is sound and by printing the leg_no immediately after this section, I can see values which appear at first inspection to be correct
However, when added to the rest of the code, it causes subsequent sections where more data is inserted using the cursor to fail with this error -
raise errors.InternalError("Unread result found.")
mysql.connector.errors.InternalError: Unread result found.
The issue seems similar to MySQL Unread Result with Python
Is the query too complex and needs splitting or is there another issue?
If the query is indeed too complex, can anyone advise how best to split this?
EDIT As per #Gord's help, Ive tried to dump any unread results
cursor.execute(query,(leg_travel_mode, leg_Orig_lat, leg_Orig_lng, leg_Dest_lat, leg_Dest_lng))
leg_no = cursor.fetchone()[0]
try:
cursor.fetchall()
except mysql.connector.errors.InterfaceError as ie:
if ie.msg == 'No result set to fetch from.':
pass
else:
raise
cursor.execute(query,(leg_travel_mode, leg_Orig_lat, leg_Orig_lng, leg_Dest_lat, leg_Dest_lng, time_stamp))
But, I still get
raise errors.InternalError("Unread result found.")
mysql.connector.errors.InternalError: Unread result found.
[Finished in 3.3s with exit code 1]
scratches head
EDIT 2 - when I print the ie.msg, I get -
No result set to fetch from
All that was required was for buffered to be set to true!
cursor = cnx.cursor(buffered=True)
The reason is that without a buffered cursor, the results are "lazily" loaded, meaning that "fetchone" actually only fetches one row from the full result set of the query. When you will use the same cursor again, it will complain that you still have n-1 results (where n is the result set amount) waiting to be fetched. However, when you use a buffered cursor the connector fetches ALL rows behind the scenes and you just take one from the connector so the mysql db won't complain.
I was able to recreate your issue. MySQL Connector/Python apparently doesn't like it if you retrieve multiple rows and don't fetch them all before closing the cursor or using it to retrieve some other stuff. For example
import mysql.connector
cnxn = mysql.connector.connect(
host='127.0.0.1',
user='root',
password='whatever',
database='mydb')
crsr = cnxn.cursor()
crsr.execute("DROP TABLE IF EXISTS pytest")
crsr.execute("""
CREATE TABLE pytest (
id INT(11) NOT NULL AUTO_INCREMENT,
firstname VARCHAR(20),
PRIMARY KEY (id)
)
""")
crsr.execute("INSERT INTO pytest (firstname) VALUES ('Gord')")
crsr.execute("INSERT INTO pytest (firstname) VALUES ('Anne')")
cnxn.commit()
crsr.execute("SELECT firstname FROM pytest")
fname = crsr.fetchone()[0]
print(fname)
crsr.execute("SELECT firstname FROM pytest") # InternalError: Unread result found.
If you only expect (or care about) one row then you can put a LIMIT on your query
crsr.execute("SELECT firstname FROM pytest LIMIT 0, 1")
fname = crsr.fetchone()[0]
print(fname)
crsr.execute("SELECT firstname FROM pytest") # OK now
or you can use fetchall() to get rid of any unread results after you have finished working with the rows you retrieved.
crsr.execute("SELECT firstname FROM pytest")
fname = crsr.fetchone()[0]
print(fname)
try:
crsr.fetchall() # fetch (and discard) remaining rows
except mysql.connector.errors.InterfaceError as ie:
if ie.msg == 'No result set to fetch from.':
# no problem, we were just at the end of the result set
pass
else:
raise
crsr.execute("SELECT firstname FROM pytest") # OK now
cursor.reset() is really what you want.
fetchall() is not good because you may end up moving unnecessary data from the database to your client.
The problem is about the buffer, maybe you disconnected from the previous MySQL connection and now it cannot perform the next statement. There are two ways to give the buffer to the cursor. First, only to the particular cursor using the following command:
import mysql.connector
cnx = mysql.connector.connect()
# Only this particular cursor will buffer results
cursor = cnx.cursor(buffered=True)
Alternatively, you could enable buffer for any cursor you use:
import mysql.connector
# All cursors created from cnx2 will be buffered by default
cnx2 = mysql.connector.connect(buffered=True)
cursor = cnx.cursor()
In case you disconnected from MySQL, the latter works for you.
Enjoy coding
If you want to get only one result from a request, and want after to reuse the same connexion for other requests, limit your sql select request to 1 using "limit 1" at the end of your request.
ex "Select field from table where x=1 limit 1;"
This method is faster using "buffered=True"
Set the consume_results argument on the connect() method to True.
cnx = mysql.connector.connect(
host="localhost",
user="user",
password="password",
database="database",
consume_results=True
)
Now instead of throwing an exception, it basically does fetchall().
Unfortunately this still makes it slow, if you have a lot of unread rows.
There is also a possibility that your connection to MySQL Workbench is disconnected. Establish the connection again. This solved the problem for me.
cursor.reset()
and then create tables and load entries
Would setting the cursor within the for loop, executing it, and then closing it again in the loop help?
Like:
for steps in result['routes'][0]['legs'][0]['steps']:
cursor = cnx.cursor()
....
leg_no = cursor.fetchone()[0]
cursor.close()
print(leg_no)
I am trying to update the data from 'Active' to 'Retired by loop through a list of devices from the specific text file.
Somehow, however, it does not filter the list of devices from the text file and update the corresponding data, making no changes to the database at all.
Could it have something to do with my for statement, or mysql statement that I came up with? Regardless of how many times I fix MYSQL, it still results the same.
What could be the problem?
Please take a look at the code below and see if there is any mistake I have made with regards to MYSQL-wise or Python-wise.
Thank you in advance for your great help. Much appreciated.
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=############;'
'Database=########;'
'Trusted_Connection=yes;')
cursor = conn.cursor()
cursor.execute('SELECT id, device_id, model_number, serial_number_1,\
status_1, user_name_1 FROM [Footprint].[fpscdb001_cmdb_004].[desktop]')
results = []
with open('H:\list.txt') as inputfile:
results = inputfile.read().splitlines()
SQL = """UPDATE [Footprint].[fpscdb001_cmdb_004].[desktop]
SET status_1 = "Retired"
WHERE device_id == %s"""
try:
for i in results:
cursor.execute(SQL, results[i])
cursor.commit()
# print(rowcount)
except:
conn.rollback()
finally:
conn.close()
It looks like the problem is both your SQL and your Python.
There is a problem with your SQL at this part: WHERE device_id == %s. In SQL, there is no ==. Instead, you use a single = to both set and check values. You should use WHERE device_id = ?.
In addition, you're using %s as a placeholder in your query. I'm not familiar with pyodbc, but a quick check of the docs looks like you should be using the ? as a placeholder.
So try this:
SQL = """UPDATE [Footprint].[fpscdb001_cmdb_004].[desktop]
SET status_1 = "Retired"
WHERE device_id = ?"""
Building on the answer that #RToyo wrote, you may be able to do this a little more quickly
we can build a list of "?" placeholders in the SQL, and then pass each item safely to the ODBC holder, using the * notation to explode the array of device id's into the ODBC execute() function. This allows you to both execute only one query, and do it securely, too
import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=############;'
'Database=########;'
'Trusted_Connection=yes;')
cursor = conn.cursor()
cursor.execute('SELECT id, device_id, model_number, serial_number_1,\
status_1, user_name_1 FROM [Footprint].[fpscdb001_cmdb_004].[desktop]')
results = []
with open('H:\list.txt') as inputfile:
results = inputfile.read().splitlines()
SQL = """UPDATE [Footprint].[fpscdb001_cmdb_004].[desktop]
SET status_1 = "Retired"
WHERE device_id in ({})""".format(("?, " * len(results))[0:-2])
try:
if len(results) > 0:
cursor.execute(SQL, *results)
except:
conn.rollback()
finally:
conn.close()
Hope this helps someone.
I am accessing a MS Access Database in Python 3.6 using pyodbc library. I can read a table, no problems. The I created a simple table (Employee). I inserted records. I was able to fetch the records too by reading the table, no problems.
I also listed the tables in the MS Access DB. Employee table shows in the list.
But when I open up the MS Access Database, I do not find the table. I changed MS Access DB to show hidden and system objects. Employee table doesn't show up.
What am I doing wrong?
Thanks
Here is the code:
import pyodbc
db_file = r'''C:\TickData2018\StooqDataAnalysis.accdb'''
user = 'admin'
password = ''
odbc_conn_str = 'DRIVER={Microsoft Access Driver (*.accdb)};DBQ=%s;UID=%s;PWD=%s' %\
(db_file, user, password)
# Or, for newer versions of the Access drivers:
odbc_conn_str = 'DRIVER={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=%s;UID=%s;PWD=%s' %\
(db_file, user, password)
conn = pyodbc.connect(odbc_conn_str)
print("connection made")
c = conn.cursor()
c.execute("SELECT * FROM 5MtsBaseForAnalysisSorted")
list1 = c.fetchmany(2)
print(list1[0][0])
print(list1[0][1])
print(list1[0][2])
try:
c.execute("""CREATE TABLE employee(
first text,
last text,
pay integer
);""")
except Exception as e:
print(e)
conn.commit
c.execute("INSERT INTO employee VALUES ('Krishna', 'Sundar', 50000)")
c.execute("INSERT INTO employee VALUES ('Divya', 'Sundar', 70000)")
c.execute("INSERT INTO employee VALUES ('Panka', 'Sundar', 70000)")
conn.commit
c.execute("SELECT * FROM employee")
print(c.fetchall())
c.tables()
rows = c.fetchall()
for row in rows:
print(row)
c.close()
del c
conn.close()
This is a general Python object model where you need to call the actual function and not its bounded name. Specifically, your commit lines are not correct where
conn.commit
Should be with open/close parentheses:
conn.commit()
Another way to see the difference is by reviewing the object's type:
type(conn.commit)
# <built-in method commit of pyodbc.Connection object at 0x000000000B772E40>
type(conn.commit())
# NoneType
I did reproduce your issue with exact code and adding parentheses resolved the issue.
An additional solution to manually committing is to set autocommit = True when the connection instance is created.
Eg:
conn = pyodbc.connect(odbc_conn_str, autocommit = True)
I am inserting a couple thousand records into a table via the python code below:
values = ''
for row in cursor:
values = values + "(" + self.quoted_comma_separate(row) + "),"
values = values[:-1]
insert_statement = "INSERT INTO t1 ({0}) VALUES {1};".format(
self.comma_separate(members), values)
db = Database()
conn = db.get_db()
cursor = conn.cursor()
cursor.execute(insert_statement)
conn.commit()
conn.close()
When I check the database after it runs none of the records show up in the database. If I go into an MySQL editor and manually commit the transaction all of the records appear. Why is my conn.commit() not working?
The insert statements were fine. Turns out I had another database connection open and it was getting confused and committing to the wrong connection or something like that. Sorry for the pointless question :)
PyMySQL, a python package to access MySQL database, seems not support SELECT ... FOR UPDATE.
In the code below, I used SELECT...FOR UPDATE to read some_table in function f(), used UPDATE to modify the table in g(), and spawned 50 threads for each function.
I expected deadlock to happen since SELECT...FOR UPDATE should block the threads spawned by g. But actually no deadlock happened. Can some one explain why?
from threading import Thread
import pymysql
def f():
db = pymysql.connect("localhost", "tester","pwd", "testDB")
cur = db.cursor()
sql = "SELECT * FROM some_table FOR UPDATE"
try:
cur.execute(sql)
except:
print("Exception in select")
def g():
db = pymysql.connect("localhost", "tester", "pwd","testDB")
cur = db.cursor()
sql = "UPDATE some_table SET val=20 WHERE id=2"
try:
cur.execute(sql)
db.commit()
except:
print("Exception in update")
db.rollback()
for _ in range(50):
Thread(target=f, args=()).start()
Thread(target=g, args=()).start()
I am using Python 3.4 and PyMySQL 0.6.6. Thank you in advance.
PyMySQL does support SELECT ... FOR UPDATE
But you need start the transaction using connection.begin()
Here's an example:
connection= pymysql.connect("localhost", "tester", "pwd","testDB")
connection.begin()
cursor = db.cursor()
cursor.execute("SELECT * FROM some_table FOR UPDATE")
Table/row (depending upon your select query) is now in locked state. Only the current connection can make changes.
To release the lock. You can,
commit the changes (if any)
connection.commit()
close the connection
connection.close()