I would like to get the result of the fetchall operation in a list instead of tuple of tuple or tuple of dictionaries.
For example,
cursor = connection.cursor() #Cursor could be a normal cursor or dict cursor
query = "Select id from bs"
cursor.execute(query)
row = cursor.fetchall()
Now, the problem is the resultant row is either ((123,),(234,)) or ({'id':123}, {'id':234})
What I am looking for is (123,234) or [123,234]. Be best if I can save on parsing the resulset.
And what about list comprehensions? If result is ((123,), (234,), (345,)):
>>> row = [item[0] for item in cursor.fetchall()]
>>> row
[123, 234, 345]
If result is ({'id': 123}, {'id': 234}, {'id': 345}):
>>> row = [item['id'] for item in cursor.fetchall()]
>>> row
[123, 234, 345]
I'm sure that after all this time, you've solved this problem, however, for some people who may not know how to get the values of a cursor as a dictionary using MySQLdb, you can use this method found here:
import MySQLdb as mdb
con = mdb.connect('localhost', 'testuser', 'test623', 'testdb')
with con:
cur = con.cursor(mdb.cursors.DictCursor)
cur.execute("SELECT * FROM Writers LIMIT 4")
rows = cur.fetchall()
for row in rows:
print row["Id"], row["Name"]
This old Q comes up on Google while searching for flattening db queries, so here are more suggestions...
Consider a fast list-flattening iterator.
Others answers use fetchall() which first loads all rows in memory, then iterates over that to make a new list. Could be inefficient. Could combine with MySQL so-called server side cursor:
# assume mysql on localhost with db test and table bs
import itertools
import MySQLdb
import MySQLdb.cursors
conn = MySQLdb.connect(host='localhost',db='test',
cursorclass=MySQLdb.cursors.SSCursor )
cursor = conn.cursor()
# insert a bunch of rows
cursor.executemany('INSERT INTO bs (id) VALUES (%s)',zip(range(1,10000)) )
conn.commit()
# retrieve and listify
cursor.execute("select id from bs")
list_of_ids = list(itertools.chain.from_iterable(cursor))
len(list_of_ids)
#9999
conn.close()
But the question is also tagged Django, which has a nice single field query flattener
class Bs(models.Model):
id_field = models.IntegerField()
list_of_ids = Bs.objects.values_list('id_field', flat=True)
Make your cursor object in this manner:
db = MySQLdb.connect("IP", "user", "password", "dbname")
cursor = db.cursor(MySQLdb.cursors.DictCursor)
Then when you perform cursor.fetchall() on a query, a tuple of dictionaries will be obtained, which you can later convert to a list.
data = cursor.fetchall()
data = list(data)
list= [list[0] for list in cursor.fetchall()]
this will render results in one list like - list = [122,45,55,44...]
If there is only one field, i can use this to make a list from database:
def getFieldAsList():
kursor.execute("Select id from bs")
id_data = kursor.fetchall()
id_list = []
for index in range(len(id_data)):
id_list.append(id_data[index][0])
return id_list
cursor.execute("""Select * From bs WHERE (id = %s)""",(id))
cursor.fetchall()
Related
I have created a database table called fruits with 3 columns, id(int primary key), fruits(text) and weight(float).
id
fruit
weight
1
Apple
80.5
2
Pear
150.8
3
Kiwi
69
How do I create a dictionary and add all the fruits and weight as key-value pairs to the dictionary?
import sqlite3
conn = sqlite3.connect("database.db")
sql = """SELECT fruit,weight FROM fruits"""
cursor = conn.execute(sql)
data = cursor.fetchall()
fruitweight= {}
i = 0
while(i < len(data)):
fruitweight['<fruitname>'] = <fruitweight>
i = i+1
Something like this:
for row in data:
name, weight = row
fruitweight[name] = weight
fetchall() returns a list of tuples of values from the database according to your query. The above unpacks the tuples into two variables.
If you want to get fancy, you can use a dictionary comprehension with tuple unpacking:
fruitweight = {name: weight for name, weight in data}
And finally, execute() actually returns a cursor that you can iterate over, so I think your code can be reduced to the following:
import sqlite3
conn = sqlite3.connect("database.db")
sql = """SELECT fruit,weight FROM fruits"""
fruitweight = {name: weight for name, weight in conn.execute(sql)}
I don't think that the cursor is correct but
you should use for loop instead:
import sqlite3
conn = sqlite3.connect("database.db")
cur = conn.cursor()
sql = """SELECT fruit,weight FROM fruits"""
cur.execute(sql)
data = cur.fetchall()
fruitweight= {}
for i in data:
fruitweight[i[0]] = i[1]
conn.close()
print(fruitweight)
You could do it like this:
import sqlite3
with sqlite3.connect('/Users/andy/andy') as conn:
cursor = conn.execute('select fruit, weight from fruits')
D = dict()
for row in cursor.fetchall():
D[row[0]] = row[1]
print(D)
When I retrieve data from my database, it is returned in the following format however I need it simply as just the text on its own:
[(u'milk',)]
[(7,)]
i have tried converting it into a string etc so that I could use a for loop to iterate through it and pop off the unneeded characters but nothing has worked
Here is my code:
def retShopping(db):
item = []
quan = []
with sqlite3.connect(db) as db:
cursor = db.cursor()
cursor.execute('''SELECT quantity FROM Shopping''')
hold = str(cursor.fetchall())
quan.append(hold)
cursor.execute('''SELECT item FROM Shopping''')
hold2 = str(cursor.fetchall())
item.append(hold2)
print(item[0])
print(quan[0])
I am hoping to be able to just end up with the strings'milk' and '7' so that I am able to use them in print statements.
try this:
def retShopping(db):
with sqlite3.connect(db) as db:
cursor = db.cursor()
cursor.execute('''SELECT quantity FROM Shopping''')
hold = cursor.fetchall()
print(hold[0][0])
cursor.execute('''SELECT item FROM Shopping''')
hold2 = cursor.fetchall()
print(hold2[0][0])
here's a run down of what I'd like to do: I have a list of table names, and I want to run sql against an oracle database and pull back the table name and row count for every table in my table list. However, not every table name in my list of table names is necessarily actually in the database. This causes my code to throw a database error. What I would like to do, is whenever I come to a table name that is not in the database, I create a dataframe that contains the table name and instead of count(*), there's some text that says 'table not found', or something similar. At the end of the loop I'm concatenating all of the dataframes into one dataframe. The overall goal here is to validate that certain tables exist and that they have the expected row counts.
query_list=[]
df_List=[]
connstr= '%s/%s#%s' %(username, password, server)
conn = cx_Oracle.connect(connstr)
with conn:
query_list = ["SELECT '%s' as tbl, count(*) FROM %s." %(elm, database) +elm for elm in table_list]
df_List = [pd.read_sql(elm,conn) for elm in query_list]
df = pd.concat(df_List)
Consider try/except handling to return query output or table not found output:
def get_table_count(sql, conn, elm):
try:
return pd.read_sql(sql, conn)
except:
return pd.DataFrame({'tbl': elm, 'note': 'table not found'}, index = [0])
with conn:
sql = "SELECT '{t}' as tbl, count(*) as table_count FROM {d}.{t}"
df_List = [get_table_count(sql.format(t = elm, d = database), conn, elm) \
for elm in table_list]
df = pd.concat(df_List, ignore_index = True)
Get a list of all the Table Names which are in the DB, then create a loop to query each Table to get the row count.
Here is a SQL statement to get a list of all Tables in an Oracle DB:
SQL:
SELECT DISTINCT TABLE_NAME FROM ALL_TAB_COLUMNS ORDER BY TABLE_NAME ASC;
Python (to make list of tables you want row counts for and which exist in the DB):
list(set(tables_that_exist_in_DB) - (set(tables_that_exist_in_DB) - set(list_of_tables_you_want)))
I'm using Python 2.7 and postgresql 9.1.
Trying to get dictionary from query, I've tried the code as described here:
http://wiki.postgresql.org/wiki/Using_psycopg2_with_PostgreSQL
import psycopg2
import psycopg2.extras
conn = psycopg2.connect("dbname=mydb host=localhost user=user password=password")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute ("select * from port")
type(cur.fetchall())
It is printing the next answer:
<type 'list'>
printing the item itself, show me that it is list.
The excepted answer was dictionary.
Edit:
Trying the next:
ans = cur.fetchall()[0]
print ans
print type(ans)
returns
[288, 'T', 51, 1, 1, '192.168.39.188']
<type 'list'>
Tnx a lot Andrey Shokhin ,
full answer is:
#!/var/bin/python
import psycopg2
import psycopg2.extras
conn = psycopg2.connect("dbname=uniart4_pr host=localhost user=user password=password")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute ("select * from port")
ans =cur.fetchall()
ans1 = []
for row in ans:
ans1.append(dict(row))
print ans1 #actually it's return
It's normal: when you call .fetchall() method returns list of tuples. But if you write
type(cur.fetchone())
it will return only one tuple with type:
<class 'psycopg2.extras.DictRow'>
After this you can use it as list or like dictionary:
cur.execute('SELECT id, msg FROM table;')
rec = cur.fetchone()
print rec[0], rec['msg']
You can also use a simple cursor iterator:
res = [json.dumps(dict(record)) for record in cursor] # it calls .fetchone() in loop
Perhaps to optimize it further we can have
#!/var/bin/python
import psycopg2
import psycopg2.extras
def get_dict_resultset(sql):
conn = psycopg2.connect("dbname=pem host=localhost user=postgres password=Drupal#1008")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute (sql)
ans =cur.fetchall()
dict_result = []
for row in ans:
dict_result.append(dict(row))
return dict_result
sql = """select * from tablename"""
return get_dict_resultset(sql)
If you don't want to use a psycopg2.extras.DictCursor you can create a list of dictionaries for the results using cursor.description:
# connect
connection = psycopg2.connect()
cursor = connection.cursor()
# query
cursor.execute("SELECT * FROM myTable")
# transform result
columns = list(cursor.description)
result = cursor.fetchall()
# make dict
results = []
for row in result:
row_dict = {}
for i, col in enumerate(columns):
row_dict[col.name] = row[i]
results.append(row_dict)
# display
print(result)
I use the following function fairly regularly:
def select_query_dict(connection, query, data=[]):
"""
Run generic select query on db, returns a list of dictionaries
"""
logger.debug('Running query: {}'.format(query))
# Open a cursor to perform database operations
cursor = connection.cursor()
logging.debug('Db connection succesful')
# execute the query
try:
logger.info('Running query.')
if len(data):
cursor.execute(query, data)
else:
cursor.execute(query)
columns = list(cursor.description)
result = cursor.fetchall()
logging.debug('Query executed succesfully')
except (Exception, psycopg2.DatabaseError) as e:
logging.error(e)
cursor.close()
exit(1)
cursor.close()
# make dict
results = []
for row in result:
row_dict = {}
for i, col in enumerate(columns):
row_dict[col.name] = row[i]
results.append(row_dict)
return results
In addition to just return only the query results as a list of dictionaries, I would suggest returning key-value pairs (column-name:row-value). Here my suggestion:
import psycopg2
import psycopg2.extras
conn = None
try:
conn = psycopg2.connect("dbname=uniart4_pr host=localhost user=user password=password")
with conn.cursor(cursor_factory=psycopg2.extras.DictCursor) as cursor:
cursor.execute("SELECT * FROM table")
column_names = [desc[0] for desc in cursor.description]
res = cursor.fetchall()
cursor.close()
return map(lambda x: dict(zip(column_names, x)), res))
except (Exception, psycopg2.DatabaseError) as e:
logger.error(e)
finally:
if conn is not None:
conn.close()
There is a built in solution to get your result as a collection of dictionary:
from psycopg2.extras import RealDictCursor
cur = conn.cursor(cursor_factory=RealDictCursor)
Modified from: https://www.peterbe.com/plog/from-postgres-to-json-strings, copyright 2013 Peter Bengtsson
For me when I convert the row to dictionary failed (solutions mentioned by others)and also could not use cursor factory.
I am using PostgreSQL 9.6.10, Below code worked for me but I am not sure if its the right way to do it.
def convert_to_dict(columns, results):
"""
This method converts the resultset from postgres to dictionary
interates the data and maps the columns to the values in result set and converts to dictionary
:param columns: List - column names return when query is executed
:param results: List / Tupple - result set from when query is executed
:return: list of dictionary- mapped with table column name and to its values
"""
allResults = []
columns = [col.name for col in columns]
if type(results) is list:
for value in results:
allResults.append(dict(zip(columns, value)))
return allResults
elif type(results) is tuple:
allResults.append(dict(zip(columns, results)))
return allResults
Way to use it:
conn = psycopg2.connect("dbname=pem host=localhost user=postgres,password=Drupal#1008")
cur = conn.cursor()
cur.execute("select * from tableNAme")
resultset = cursor.fetchall()
result = convert_to_dict(cursor.description, resultset)
print(result)
resultset = cursor.fetchone()
result = convert_to_dict(cursor.description, resultset)
print(result)
Contents of './config.py'
#!/usr/bin/python
PGCONF = {
"user": "postgres",
"password": "postgres",
"host": "localhost",
"database": "database_name"
}
contents of './main.py'
#!/usr/bin/python
from config import PGCONF
import psycopg2
import psycopg2.extras
# open connection
conn = psycopg2.connect(**PGCONF)
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
# declare lambda function
fetch_all_as_dict = lambda cursor: [dict(row) for row in cursor]
# execute any query of your choice
cur.execute("""select * from table_name limit 1""")
# get all rows as list of dicts
print(fetch_all_as_dict(cur))
# close cursor and connection
cur.close()
conn.close()
In one of my django views I query database using plain sql (not orm) and return results.
sql = "select * from foo_bar"
cursor = connection.cursor()
cursor.execute(sql)
rows = cursor.fetchall()
I am getting the data fine, but not the column names. How can I get the field names of the result set that is returned?
On the Django docs, there's a pretty simple method provided (which does indeed use cursor.description, as Ignacio answered).
def dictfetchall(cursor):
"Return all rows from a cursor as a dict"
columns = [col[0] for col in cursor.description]
return [
dict(zip(columns, row))
for row in cursor.fetchall()
]
According to PEP 249, you can try using cursor.description, but this is not entirely reliable.
I have found a nice solution in Doug Hellmann's blog:
http://doughellmann.com/2007/12/30/using-raw-sql-in-django.html
from itertools import *
from django.db import connection
def query_to_dicts(query_string, *query_args):
"""Run a simple query and produce a generator
that returns the results as a bunch of dictionaries
with keys for the column values selected.
"""
cursor = connection.cursor()
cursor.execute(query_string, query_args)
col_names = [desc[0] for desc in cursor.description]
while True:
row = cursor.fetchone()
if row is None:
break
row_dict = dict(izip(col_names, row))
yield row_dict
return
Example usage:
row_dicts = query_to_dicts("""select * from table""")
try the following code :
def read_data(db_name,tbl_name):
details = sfconfig_1.dbdetails
connect_string = 'DRIVER=ODBC Driver 17 for SQL Server;SERVER={server}; DATABASE={database};UID={username}\
;PWD={password};Encrypt=YES;TrustServerCertificate=YES'.format(**details)
connection = pyodbc.connect(connect_string)#connecting to the server
print("connencted to db")
# query syntax
query = 'select top 100 * from '+'[{}].[dbo].[{}]'.format(db_name,tbl_name) + ' t where t.chargeid ='+ "'622102*3'"+';'
#print(query,"\n")
df = pd.read_sql_query(query,con=connection)
print(df.iloc[0])
return "connected to db...................."