I'm creating a simple app to connect to a postgresql database, query it to list all the tables, and query each of the table to output some information. My code is running but I need to fix some issues of the part below. df_dbtables is my pandas dataframe of db schemas and tables.
for index, row in df_dbtables.iterrows():
try:
schema_table = row['schema'] + "." + row['table']
cur.execute("SELECT type,stylename FROM %s" % schema_table)
rows = cur.fetchall()
for row in rows:
data.append({"Type" : row[0], "Stylename" : row[1]})
except:
continue
Issue #1:
My first table runs perfectly. But the second table doesn't have a type field, so It runs into this postgresql error: psycopg2.errors.UndefinedColumn: column "type" does not exist Then, the code runs into the except that tells the code to continue. The problem is that after the first time running into the except, all my others table queries run into the except too, ignoring that they have type and stylename fields. How can I properly ignore this error message and continue to the next iteration? Also, what is the best way to output SQL errors using try/except?
Issue #2:
Once fixed the above issue I would like to know how can I prevent this: If a field doesn't exists and run into a SQL error, it will ignore the other field (if it exists) because it will run into the except. For example: The script is querying table X, that is a table without type field, when it runs into the except It will ignore its stylenames data.
Improvement:
I've tried many ways to parameterize the sql query. I know that way that I used is very prune to SQL injections, but the correct ways just don't work.
Tried these methods and others but couldn't run them successfully.
APP
My next step is creating a Flask app for this code. So, if you have a solution that uses Flask it will be welcome.
Code updated at July 12th. But still with the same issues:
for index, row in df_dbtables.iterrows():
try:
schema_table = row['schema'] + "." + row['table']
cur = con.cursor()
cur.execute("SELECT type,stylename FROM %s" % schema_table)
for r in rs:
data.append({"Type" : r[0], "Stylename" : r[1]})
#except psycopg2.OperationalError: traceback.print_exc()
except: continue
Related
I'm trying to fill a PostgreSQL table (psycopg2, Python) with the filenames I have in a specific folder. I have created a function that should do the trick, but I get the error:
not all arguments converted during string formatting,
when I run my function. I did a test run and called the function in the following way:
insert_file_names_into_database(["filename1_without_extension", "filename2_without_extension"]),
and I had no problems and the INSERT worked fine. If I did the following:
insert_file_names_into_database(["filename1.extension", "filename2.extension"]),
Then I get the error above. So the problem seems to be the "." character (e.g. image.jpg) which causes the SQL INSERT to fail. I tried to consult the Psycopg2 docs about this, but I found no examples relating to this specific case.
How should I edit the piece of code so I can get to work even with "." characters in the filenames?
def insert_file_names_into_database(file_name_list):
""" insert multiple filenames into the table """
sql = "INSERT INTO mytable(filename) VALUES(%s)"
conn = None
try:
# read database configuration
# connect to the PostgreSQL database
conn = psycopg2.connect(
host="localhost",
database="mydatabase",
user="myusername",
password="mypassword")
# create a new cursor
cur = conn.cursor()
# execute the INSERT statement
cur.executemany(sql, file_name_list)
# commit the changes to the database
conn.commit()
# close communication with the database
cur.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()
Solved it myself already. I knew I should be using tuples when working with the INSERT, but my function worked fine with list of strings without the "." characters.
The solution I got working was to convert the list of strings into a list of tuples like so:
tuple_file_name = [tuple((file_name,)) for file_name in file_name_list]
So for example if:
file_name_list = ["filename1.jpg", "filename2.jpg"]
Then giving this as input to my function fails. But by making it a list of tuples:
tuple_file_name = [tuple((file_name,)) for file_name in file_name_list]
print(tuple_file_name)
[('filename1.jpg',), ('filename2.jpg',)]
Then now the function accepts the input tuple_file_name and the filenames are saved into the SQL table.
I am trying to find the latest entry in a MySQL database by using a query with SELECT MAX(id). I already get the latest id, so I know the query works, but now I want to use it in a while loop so I keep getting the latest entry with each iteration.
This is what I have so far:
import pymysql
con = pymysql.connect(host='.....', user='.....',
password='.....', database='.....')
cur = con.cursor()
while True:
query = "SELECT MAX(id) FROM reports"
cur.execute(query)
data = cur.fetchall()
last = (data[0][0])
print(last)
The problem is that I keep getting the same result after updating the database. For instance, right now I have 45 entries, so my script prints '45' in a while loop. But after I add another row to the table it keeps printing '45' instead of the '46' I would expect. When I stop the script and run it again, it will print '46' and keep printing this even after I add another row.
I have only started working with MySQL about two weeks ago, so I don't have all that much knowledge about it. I feel like I'm missing something really small here. What should I do? Any help would be greatly appreciated.
I had this same problem, and just wanted to make it clear for anyone else searching for the solution.
Setting autocommit to True solved my issue and didn't require calling a commit after each query.
import pymysql
con = pymysql.connect(host='.....', user='.....',
password='.....', database='.....')
con.autocommit = True
cur = con.cursor()
while True:
query = "SELECT MAX(id) FROM reports"
cur.execute(query)
data = cur.fetchall()
last = (data[0][0])
print(last)
Here is a link to the documentation
I have written a script to help me work with a database. Specifically, I am trying to work with files on disk and add the result of this work to my database. I have copied the code below, but removed most of the logic which isn't related to my database to try to keep this question broad and helpful.
I used the code to operate on the files and add the result to the database, overwriting any files with the same identifier as the one I was working on. Later, I modified the script to ignore documents which have already been added to the database, and now whenever I run it I get an error:
pymysql.err.OperationalError: (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")
It seems like the server is rejecting the requests, possibly because I have written my code poorly? I have noticed that the error always occurs at the same place in the list of files, which doesn't change. If I re-run run the code, replacing the file list with a list of only the file on which the program crashes, it works fine. This makes me think that after making a certain number of requests, the database just bottoms out.
I'm using Python 3 and MySQL Community Edition Version 14.14 on OS X.
Code (stripped of stuff that doesn't have to do with the database):
import pymysql
# Stars for user-specific stuff
connection = pymysql.connect(host='localhost',
user='root',
password='*******',
db='*******',
use_unicode=True,
charset="utf8mb4",
)
cursor = connection.cursor()
f_arr = # An array of all of my data objects
def convertF(file_):
# General layout: Try to work with input and add it the result to DB. The work can raise an exception
# If the record already exists in the DB, ignore it
# Elif the work was already done and the result is on disk, put it on the database
# Else do the work and put it on the database - this can raise exceptions
# Except: Try another way to do the work, and put the result in the database. This can raise an error
# Second (nested) except: Add the record to the database with indicator that the work failed
# This worked before I added the initial check on whether or not the record already exists in the database. Now, for some reason, I get the error:
# pymysql.err.OperationalError: (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")
# I'm pretty sure that I have written code to work poorly with the database. I had hoped to finish this task quickly instead of efficiently.
try:
# Find record in DB, if text exists just ignore the record
rc = cursor.execute("SELECT LENGTH(text) FROM table WHERE name = '{0}'".format(file_["name"]))
length = cursor.fetchall()[0][0] # Gets the length
if length != None and length > 4:
pass
elif ( "work already finished on disk" ):
# get "result_text" from disk
cmd = "UPDATE table SET text = %s, hascontent = 1 WHERE name = %s"
cursor.execute(cmd, ( pymysql.escape_string(result_text), file_["name"] ))
connection.commit()
else:
# do work to get result_text
cmd = "UPDATE table SET text = %s, hascontent = 1 WHERE name = %s"
cursor.execute(cmd, ( pymysql.escape_string(result_text), file_["name"] ))
connection.commit()
except:
try:
# Alternate method of work to get result_text
cmd = "UPDATE table SET text = %s, hascontent = 1 WHERE name = %s"
cursor.execute(cmd, ( pymysql.escape_string(result_text), file_["name"] ))
connection.commit()
except:
# Since the job can't be done, tell the database
cmd = "UPDATE table SET text = %s, hascontent = 0 WHERE name = %s"
cursor.execute(cmd, ( "NO CONTENT", file_["name"]) )
connection.commit()
for file in f_arr:
convertF(file)
Mysql Server Has Gone Away
This problem is described extensively at http://dev.mysql.com/doc/refman/5.7/en/gone-away.html the usual cause is that the server has disconnected for whatever reason and the usual remedy is to retry the query or to reconnect and retry.
But why this breaks your code is because of the way you have written your code. See below
Possibly because I have written my code poorly?
Since you asked.
rc = cursor.execute("SELECT LENGTH(text) FROM table WHERE name = '{0}'".format(file_["name"]))
This is a bad habit. The manually explicitly warns you against doing this to avoid SQL injections. The correct way is
rc = cursor.execute("SELECT LENGTH(text) FROM table WHERE name = %s", (file_["name"],))
The second problem with the above code is that you don't need to check if a value exists before you try to update it. You can delete the above line and it's associated if else and jump straight to the update. Besides, our elif and else seem to do exactly the same thing. So your code can just be
try:
cmd = "UPDATE table SET text = %s, hascontent = 1 WHERE name = %s"
cursor.execute(cmd, ( pymysql.escape_string(result_text), file_["name"] ))
connection.commit()
except: # <-- next problem.
And we come to the next problem. Never ever catch generic exceptions like this. you should always catch specific exceptions like TypeError, AttributeError etc. When catching generic exceptions is unavoidable, you should at least log it.
For example, here you could catch connection errors and attempt to reconnect to the database. Then the code will not stop executing when your server gone away problem happens.
I've solved the same error in the case when I tried to make a bulk inserts by reducing the number of lines I wanted to insert in one command.
Even the maximum number of lines for bulk insert was much higher, I had this kind of error.
I'm trying to retrieve data from a mysql server using the Python MySQL Connector.
So far, I have the query set up right using example code I found, and other resources. The problem is, I'm not sure how get it to print back all the rows, instead of just a certain row.
The code I'm using is:
def dbConnect(tag):
qrfid = tag
cnx = mysql.connector.connect(user='root', password='root', host='localhost', database='test')
cursor = cnx.cursor(buffered=True)
query = ("SELECT * FROM `user` WHERE `rfid`= %s")
cursor.execute(query, (qrfid,))
if not cursor.rowcount:
print("No results found")
else:
for row in cursor:
print row[1]
cursor.close()
cnx.close()
I'm using a test table that has only 3 columns in it: id, name, and rfid.
The above code only prints out the 2nd column, name, where if I put row[0], I get id, etc.
I'm use to using PHP for queries, but the RFID readers I'm using only have Python, Flash, and C support. Python is the language I know the most out of those 3.
Thanks!
Morning Trevor,
it is a bit late, but your question seems to be unanswered and should not stay like this.
I guess you've mistaken the way the data is returned. You assumed that you access the data of the header using row[0] and the records using row[1], row[2] ...
But actually, the "for row in cursor:" statement creates an iterator for the cursor object, which returns one complete record per loop. The record is represented in form of a tuple, referenced by name row. You can display the while record using "print(record)" or slice out the first column with row[0], second with row[1], etc.
So, everything seems fine. However, be careful with the query where clause. When comparing to a string (%s placeholder), you should always enclose the string in quotes. Some SQL interpreters expect string literals in comparisons to be quoted.
I'm trying to run a simple insert query to a database. I have it configured correctly, and it should work, but it doesn't. For some reason I get the following error on this query:
Query:
INSERT INTO searches (query) VALUES ('test')
Error:
(1062, "Duplicate entry 'test' for key 'query'")
The query runs without problems in the MySQL console so it must be a problem with Python? Here's my Python code:
def increase_search_count(search_query):
from django.db import connection, transaction
search_query = search_query.strip()
cursor = connection.cursor()
rows = cursor.execute("INSERT INTO searches (query) VALUES ('test')")
I know there are much better ways to handle databases, but I'm new to Python, and I have a deadline. I'd just like to get this to work, I have another SELECT query in another function and that one runs without any problems!
Any ideas what might be wrong?
The way that query is constructed means you will always be inserting 'test' into the database, and seeing the query is likely the primary key in your table, it will be creating duplicate rows.
The query should be something like "INSERT INTO searches (query) VALUES ('" variable "')" so you don't insert the same value over and over.