I'm having a problem while trying to simply execute data from rows from db (sqlite3). The DB input has 4 fields, therefore once entered they're being saved. But here's my problem, where I execute all of the 4 rows, if one of the fields was not filled I get an error.
That's the database execute code:
def ids(self):
con = lite.connect('foo.db')
with con:
cur = con.cursor()
cur.execute("SELECT Id FROM foo")
while True:
ids = cur.fetchall()
if ids == None:
continue
return ids
And since there are 4 rows, my output code:
print ''.join(ids[0]) + ',' + ''.join(ids[1]) + ',' + ''.join(ids[2])
+ ',' + ''.join(ids[3])
so my question is how to make an exception when there's no existing row to not show anything and just leave the ones that actually exist? I tried doing if ids[0] is not None: #do something but that would make my code really slow and it's non-pythonic way I guess. Is there any better way to make that work? Any help will be appreciated.
You don't seem to have 4 rows. Make it generic and just join an arbitrary number of rows:
ids = someobject.ids()
print ','.join(''.join(row) for row in ids)
You can simplify your database query, there is no need to 'poll' the query:
def ids(self):
with lite.connect('foo.db') as con:
cur = con.cursor()
cur.execute("SELECT Id FROM foo")
return cur.fetchall()
You could also just loop directly over the cursor, the database will handle buffering as you fetch:
def ids(self):
with lite.connect('foo.db') as con:
cur = con.cursor()
cur.execute("SELECT Id FROM foo")
return cur # just the cursor, no fetching
ids = someobject.ids()
# this'll loop over the cursor, which yields rows as required
print ','.join(''.join(row) for row in ids)
Related
I currently have a list of id's approx. of size 10,000. I need to update all rows in the mySQL table which have an id in the inactive_ids list that you see below. I need to change their active status to 'No' which is a column in the mySQL table.
I am using mysql.connector python library.
When I run the code below, it is taking about 0.7 seconds to execute each iteration in the for loop. Thats about a 2 hour run time for all 10,000 id's to be changed. Is there a more optimal/quicker way to do this?
# inactive_ids are unique strings something like shown below
# inactive_ids = ['a9okeoko', 'sdfhreaa', 'xsdfasy', ..., 'asdfad']
# initialize connection
mydb = mysql.connector.connect(
user="REMOVED",
password="REMOVED",
host="REMOVED",
database="REMOVED"
)
# initialize cursor
mycursor = mydb.cursor(buffered=True)
# Function to execute multiple lines
def alter(state, msg, count):
result = mycursor.execute(state, multi=True)
result.send(None)
print(str(count), ': ', msg, result)
count += 1
return count
# Try to execute, throw exception if fails
try:
count = 0
for Id in inactive_ids:
# SAVE THE QUERY AS STRING
sql_update = "UPDATE test_table SET Active = 'No' WHERE NoticeId = '" + Id + "'"
# ALTER
count = alter(sql_update, "done", count)
# commits all changes to the database
mydb.commit()
except Exception as e:
mydb.rollback()
raise e
Do it with a single query that uses IN (...) instead of multiple queries.
placeholders = ','.join(['%s'] * len(inactive_ids))
sql_update = f"""
UPDATE test_table
SET Active = 'No'
WHERE NoticeId IN ({placeholders})
"""
mycursor.execute(sql_update, inactive_ids)
I write to see how I can get only one data to show by a print when I make a query in python, when I do the query it should only give me a number but I cannot show or access it.
def run_query(self, query, parameters = ()):
with sqlite3.connect(self.db_name) as conn:
cursor = conn.cursor()
result = cursor.execute(query, parameters)
conn.commit()
return result
def get_horarios(self):
query = 'SELECT hora FROM horarios where horario=1'
db_rows = self.run_query(query)
print(db_rows)
To show the first row in the results of the query:
print(db_rows.fetchone())
To show all of the results of the query:
print(db_rows.fetchall())
or
for row in db_rows.fetchall():
print(row)
The query always return a list. To access the first item, you can do:
print(db_rows[0])
I am trying to print SQL result through python code, where I an trying to pass different predicates of the where clause from a for loop. But the code only taking the last value from the loop and giving the result.
In the below example I have two distinct id values 'aaa' and 'bbb'. There are 4 records for id value = 'aaa' and 2 records for the id value = 'bbb'.
But the below code only giving me the result for the id value ='bbb' not for id value 'aaa'
Can anyone help to identify what exactly wrong I am doing?
import pymysql
db = pymysql.connect(host="localhost", user="user1", passwd="pass1", db="db1")
cur = db.cursor()
in_lst=['aaa', 'bbb']
for i in in_lst:
Sql = "SELECT id, val, typ FROM test123 Where id='{inpt}'".format(inpt=i)
print(Sql)
cur.execute(Sql)
records = cur.fetchall()
print(records)
db.close()
The result I am getting as below
C:\Python34\python.exe C:/Users/Koushik/PycharmProjects/Test20161204/20170405.py
SELECT id, val, typ FROM test123 Where id='bbb'
(('bbb', 5, '1a'), ('bbb', 17, '1d'))
Process finished with exit code 0
import pymysql
db = pymysql.connect(host="localhost", user="root", passwd="1234", db="sakila")
cur = db.cursor()
in_lst=['1', '2']
for i in in_lst:
Sql = "SELECT * FROM actor Where actor_id='{inpt}'".format(inpt=i)
print(Sql)
cur.execute(Sql)
records = cur.fetchall()
print(records)
db.close()
Indentation is your problem, please update the code according to your needs...
Within your for loop, you're formatting the sql statement to replace "{inpt}" with "aaa". However, before you do anything with that value, you're immediately overwriting it with the "bbb" version.
You would need to either:
Store the results somehow before the next iteration of the loop, then process them outside of the loop.
Process the results within the loop.
Something like the following will give you a list containing both results from the fetchall() calls:
import pymysql
db = pymysql.connect(host="localhost", user="user1", passwd="pass1", db="db1")
cur = db.cursor()
in_lst=['aaa', 'bbb']
records = list()
for i in in_lst:
Sql = "SELECT id, val, typ FROM test123 Where id='{inpt}'".format(inpt=i)
print(Sql)
cur.execute(Sql)
records.append(cur.fetchall())
print(records)
db.close()
Issue: I can't figure out how to run a query in the correct way so that it returns a mapped dictionary. The query will use counts from multiple tables.
I am using psycopg2 for a postgresql database, and I will be using the results to create a report on day to day deltas on these counts.
Given that, can someone provide an example on how to execute multiple queries and return a dictionary that I can use for comparison purposes? Thanks! I image in a for loop is needed somewhere in here.
tables = ['table1', 'table2']
def db_query():
query = "select count(*) from (a_table) where error_string != '';"
conn = psycopg2.connect(database=db, user=user, password=password, host=host)
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute(query, tables)
output = cur.fetchall()
conn.close()
return output
I haven't used postgresql, so you might want to also check this out as a reference: How to store count values in python.
That being said, rearrange your code into something like this. Be sure to make conn global so you don't have to make more than one connection, and make sure you're also closing cur:
conn = None
def driverFunc():
global conn
try:
conn = psycopg2.connect(database=db, user=user, password=password, host=host)
tables = ['table1', 'table2']
countDict = {}
for thisTable in tables:
db_query(thisTable, countDict)
finally:
if not conn == None:
conn.close()
def db_query(tableName, countDict):
# Beware of SQL injection with the following line:
query = "select count(*) from " + tableName + " where error_string != '';"
cur = None
try:
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute(query)
countDict[tableName] = int(cur.fetchone())
finally:
if not cur == None:
cur.close()
I have this code:
dsn = cx_Oracle.makedsn(hostname, port, sid)
orcl = cx_Oracle.connect(username + '/' + password + '#' + dsn)
curs = orcl.cursor()
sql = "select TEMPLATE from my_table where id ='6'"
curs.execute(sql)
rows = curs.fetchall()
print rows
template = rows[0][0]
orcl.close()
print template.read()
When I do print rows, I get this:
[(<cx_Oracle.LOB object at 0x0000000001D49990>,)]
However, when I do print template.read(), I get this error:
cx_Oracle.DatabaseError: Invalid handle!
Do how do I get and read this data? Thanks.
I've found out that this happens in case when connection to Oracle is closed before the cx_Oracle.LOB.read() method is used.
orcl = cx_Oracle.connect(usrpass+'#'+dbase)
c = orcl.cursor()
c.execute(sq)
dane = c.fetchall()
orcl.close() # before reading LOB to str
wkt = dane[0][0].read()
And I get: DatabaseError: Invalid handle!
But the following code works:
orcl = cx_Oracle.connect(usrpass+'#'+dbase)
c = orcl.cursor()
c.execute(sq)
dane = c.fetchall()
wkt = dane[0][0].read()
orcl.close() # after reading LOB to str
Figured it out. I have to do something like this:
curs.execute(sql)
for row in curs:
print row[0].read()
You basically have to loop through the fetchall object
dsn = cx_Oracle.makedsn(hostname, port, sid)
orcl = cx_Oracle.connect(username + '/' + password + '#' + dsn)
curs = orcl.cursor()
sql = "select TEMPLATE from my_table where id ='6'"
curs.execute(sql)
rows = curs.fetchall()
for x in rows:
list_ = list(x)
print(list_)
There should be an extra comma in the for loop, see in below code, i have supplied an extra comma after x in for loop.
dsn = cx_Oracle.makedsn(hostname, port, sid)
orcl = cx_Oracle.connect(username + '/' + password + '#' + dsn)
curs = orcl.cursor()
sql = "select TEMPLATE from my_table where id ='6'"
curs.execute(sql)
rows = curs.fetchall()
for x, in rows:
print(x)
I had the same problem with in a slightly different context. I needed to query a +27000 rows table and it turns out that cx_Oracle cuts the connection to the DB after a while.
While a connection to the db is open, you can use the read() method of the cx_Oracle.Lob object to transform it into a string. But if the query brings a table that is too big, it won´t work because the connection will stop at some point and when you want to read the results from the query you´ll gt an error on the cx_Oracle objects.
I tried many things, like setting
connection.callTimeout = 0 (according to documentation, this means it would wait indefinetly), using fetchall() and then putting the results on a dataframe or numpy array but I could never read the cx_Oracle.Lob objects.
If I try to run the query using pandas.DataFrame.read_sql(query, connection) The dataframe would contain cx_Oracle.Lob objects with the connection closed, making them useless. (Again this only happens if the table is very big)
In the end I found a way of getting around this by querying and creating a csv file inmediatlely after, even though I know it´s not ideal.
def csv_from_sql(sql: str, path: str="dataframe.csv") -> bool:
try:
with cx_Oracle.connect(config.username, config.password, config.database, encoding=config.encoding) as connection:
connection.callTimeout = 0
data = pd.read_sql(sql, con=connection)
data.to_csv(path)
print("FILE CREATED")
except cx_Oracle.Error as error:
print(error)
return False
finally:
print("PROCESS ENDED\n")
return True
def make_query(sql: str, path: str="dataframe.csv") -> pd.DataFrame:
if csv_from_sql(sql, path):
dataframe = pd.read_csv("dataframe.csv")
return dataframe
return pd.DataFrame()
This took a long time (about 4 to 5 minutes) to bring my +27000-rows table, but it worked when everything else didn´t.
If anyone knows a better way, it would be helpful for me too.