Is there any way to get the results from a fetchall() as a dictionary using pymysql?
PyMySQL includes a DictCursor. It does what I think you want. Here's how to use it:
import pymysql
import pymysql.cursors
connection = pymysql.connect(db="test")
cursor = connection.cursor(pymysql.cursors.DictCursor)
cursor.execute("SELECT ...")
https://github.com/PyMySQL/PyMySQL/blob/master/pymysql/tests/test_DictCursor.py
Use pymysql.cursors.DictCursor, which will return rows represented as dictionaries mapping column names to values.
A few ways to use it...
Create a connection object and have all cursors spawned from it be DictCursors:
>>> import pymysql
>>> connection = pymysql.connect(db='foo', cursorclass=pymysql.cursors.DictCursor)
>>> with connection.cursor() as cursor:
... print cursor
...
<pymysql.cursors.DictCursor object at 0x7f87682fefd0>
>>> with connection.cursor() as cursor:
... cursor.execute("SELECT * FROM bar")
... print cursor.fetchall()
...
2
[{u'col2': 'rty', u'col1': 'qwe'}, {u'col2': 'fgh', u'col1': 'asd'}]
Create a DictCursor from an ordinary connection object:
>>> connection = pymysql.connect(db='foo')
>>> with connection.cursor(pymysql.cursors.DictCursor) as cursor:
... print cursor
...
<pymysql.cursors.DictCursor object at 0x7f876830c050>
Connect and create a DictCursor in one line with with:
>>> from pymysql.cursors import DictCursor
>>> with pymysql.connect(db='foo', cursorclass=DictCursor) as cursor:
... print cursor
...
<pymysql.cursors.DictCursor object at 0x7f8767769490>
Use a DictCursor in the cursor() method.
If you mean that you want to fetch two columns, and return them as a dictionary, you can use this method.
def fetch_as_dict(cursor select_query):
'''Execute a select query and return the outcome as a dict.'''
cursor.execute(select_query)
data = cursor.fetchall()
try:
result = dict(data)
except:
msg = 'SELECT query must have exactly two columns'
raise AssertionError(msg)
return result
Related
How to convert
cursor.execute("SELECT strftime('%m.%d.%Y %H:%M:%S', timestamp, 'localtime'), temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
for row in results[:-1]:
row=results[-1]
rowstr="['{0}',{1}]\n".format(str(row[0]),str(row[1]))
temp_chart_table+=rowstr
result
['01.15.2015 21:38:52',21.812]
into dictionary output in form of:
[{timestamp:'01.15.2015 21:38:52',temp:21.812}]
Edit
This is fetchone sample I currenyly use and it works fine:
def get_avg():
conn=sqlite3.connect(dbname)
curs=conn.cursor()
curs.execute("SELECT ROUND(avg(temp), 2.2) FROM data WHERE timestamp>datetime('now','-1 hour') AND timestamp<=datetime('now')")
rowavg=curs.fetchone()
#print rowavg
#rowstrmin=format(str(rowavg[0]))
#return rowstrmin
**d = [{"avg":rowavg[0]}]**
return d
conn.close()
#print get_avg()
schema = {"avg": ("number", "avg")}
data = get_avg()
# Loading it into gviz_api.DataTable
data_table = gviz_api.DataTable(schema)
data_table.LoadData(data)
json = data_table.ToJSon()
#print results
#print "Content-type: application/json\n\n"
print "Content-type: application/json"
print
print json
Then I make jQuery call and pass it into javascript and found help for that in here
ajax json query directly to python generated html gets undefined
As I can see you are using format to write in the form of a string.
Note from the docs
it is not possible to use { and } as fill char while using the str.format() method
To make it look like a dictionary you can do
"[{timestamp:'%s',temp:%s}]\n"%(str(row[0]),str(row[1]))
But if you want to make it a dictionary then you will have to do
row_dic = [{'timestamp':row[0],'temp':row[1]}]
Try this instead:
cursor.execute("SELECT strftime('%m.%d.%Y %H:%M:%S', timestamp, 'localtime'), temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
temp_chart_table = []
for row in results:
temp_chart_table.append({'timestamp': row[0], 'temp': row[1]})
In most of the python database adapters you can use a DictCursor to retrieve records using an interface similar to the Python dictionaries instead of the tuples.
Using psycopg2:
>>> dict_cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
>>> dict_cur.execute("INSERT INTO test (num, data) VALUES(%s, %s)",
... (100, "abc'def"))
>>> dict_cur.execute("SELECT * FROM test")
>>> rec = dict_cur.fetchone()
>>> rec['id']
1
>>> rec['num']
100
>>> rec['data']
"abc'def"
Using MySQLdb:
>>> import MySQLdb
>>> import MySQLdb.cursors
>>> myDb = MySQLdb.connect(user='andy47', passwd='password', db='db_name', cursorclass=MySQLdb.cursors.DictCursor)
>>> myCurs = myDb.cursor()
>>> myCurs.execute("SELECT columna, columnb FROM tablea")
>>> firstRow = myCurs.fetchone()
{'columna':'first value', 'columnb':'second value'}
def stuffToDict(stuff):
return {"timestamp":stuff[0],"temp":stuff[1]}
That would be a dictionary. The sample output you showed is a list of dictionaries, which can be achieved by putting square brackets around the dictionary. I don't know why you'd want that, though. Also, because of the missing quotes, it wasn't legal python syntax.
Use MySQLdb's cursor library.
import MySQLdb
import MySQLdb.cursors
conn = MySQLdb.connect(host=db_host, user=db_user, passwd=db_passwd, db=db_schema, port=db_port, cursorclass=MySQLdb.cursors.DictCursor)
cursor = conn.cursor()
cursor.execute("SELECT timestamp, localtime, temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
Then you have access to the results as a dictionary:
>>> results['timestamp']
14146587
>>> results['localtime']
20:08:07
>>> results['temp']
temp_variable_whatever
I'm using Python 2.7 and postgresql 9.1.
Trying to get dictionary from query, I've tried the code as described here:
http://wiki.postgresql.org/wiki/Using_psycopg2_with_PostgreSQL
import psycopg2
import psycopg2.extras
conn = psycopg2.connect("dbname=mydb host=localhost user=user password=password")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute ("select * from port")
type(cur.fetchall())
It is printing the next answer:
<type 'list'>
printing the item itself, show me that it is list.
The excepted answer was dictionary.
Edit:
Trying the next:
ans = cur.fetchall()[0]
print ans
print type(ans)
returns
[288, 'T', 51, 1, 1, '192.168.39.188']
<type 'list'>
Tnx a lot Andrey Shokhin ,
full answer is:
#!/var/bin/python
import psycopg2
import psycopg2.extras
conn = psycopg2.connect("dbname=uniart4_pr host=localhost user=user password=password")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute ("select * from port")
ans =cur.fetchall()
ans1 = []
for row in ans:
ans1.append(dict(row))
print ans1 #actually it's return
It's normal: when you call .fetchall() method returns list of tuples. But if you write
type(cur.fetchone())
it will return only one tuple with type:
<class 'psycopg2.extras.DictRow'>
After this you can use it as list or like dictionary:
cur.execute('SELECT id, msg FROM table;')
rec = cur.fetchone()
print rec[0], rec['msg']
You can also use a simple cursor iterator:
res = [json.dumps(dict(record)) for record in cursor] # it calls .fetchone() in loop
Perhaps to optimize it further we can have
#!/var/bin/python
import psycopg2
import psycopg2.extras
def get_dict_resultset(sql):
conn = psycopg2.connect("dbname=pem host=localhost user=postgres password=Drupal#1008")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute (sql)
ans =cur.fetchall()
dict_result = []
for row in ans:
dict_result.append(dict(row))
return dict_result
sql = """select * from tablename"""
return get_dict_resultset(sql)
If you don't want to use a psycopg2.extras.DictCursor you can create a list of dictionaries for the results using cursor.description:
# connect
connection = psycopg2.connect()
cursor = connection.cursor()
# query
cursor.execute("SELECT * FROM myTable")
# transform result
columns = list(cursor.description)
result = cursor.fetchall()
# make dict
results = []
for row in result:
row_dict = {}
for i, col in enumerate(columns):
row_dict[col.name] = row[i]
results.append(row_dict)
# display
print(result)
I use the following function fairly regularly:
def select_query_dict(connection, query, data=[]):
"""
Run generic select query on db, returns a list of dictionaries
"""
logger.debug('Running query: {}'.format(query))
# Open a cursor to perform database operations
cursor = connection.cursor()
logging.debug('Db connection succesful')
# execute the query
try:
logger.info('Running query.')
if len(data):
cursor.execute(query, data)
else:
cursor.execute(query)
columns = list(cursor.description)
result = cursor.fetchall()
logging.debug('Query executed succesfully')
except (Exception, psycopg2.DatabaseError) as e:
logging.error(e)
cursor.close()
exit(1)
cursor.close()
# make dict
results = []
for row in result:
row_dict = {}
for i, col in enumerate(columns):
row_dict[col.name] = row[i]
results.append(row_dict)
return results
In addition to just return only the query results as a list of dictionaries, I would suggest returning key-value pairs (column-name:row-value). Here my suggestion:
import psycopg2
import psycopg2.extras
conn = None
try:
conn = psycopg2.connect("dbname=uniart4_pr host=localhost user=user password=password")
with conn.cursor(cursor_factory=psycopg2.extras.DictCursor) as cursor:
cursor.execute("SELECT * FROM table")
column_names = [desc[0] for desc in cursor.description]
res = cursor.fetchall()
cursor.close()
return map(lambda x: dict(zip(column_names, x)), res))
except (Exception, psycopg2.DatabaseError) as e:
logger.error(e)
finally:
if conn is not None:
conn.close()
There is a built in solution to get your result as a collection of dictionary:
from psycopg2.extras import RealDictCursor
cur = conn.cursor(cursor_factory=RealDictCursor)
Modified from: https://www.peterbe.com/plog/from-postgres-to-json-strings, copyright 2013 Peter Bengtsson
For me when I convert the row to dictionary failed (solutions mentioned by others)and also could not use cursor factory.
I am using PostgreSQL 9.6.10, Below code worked for me but I am not sure if its the right way to do it.
def convert_to_dict(columns, results):
"""
This method converts the resultset from postgres to dictionary
interates the data and maps the columns to the values in result set and converts to dictionary
:param columns: List - column names return when query is executed
:param results: List / Tupple - result set from when query is executed
:return: list of dictionary- mapped with table column name and to its values
"""
allResults = []
columns = [col.name for col in columns]
if type(results) is list:
for value in results:
allResults.append(dict(zip(columns, value)))
return allResults
elif type(results) is tuple:
allResults.append(dict(zip(columns, results)))
return allResults
Way to use it:
conn = psycopg2.connect("dbname=pem host=localhost user=postgres,password=Drupal#1008")
cur = conn.cursor()
cur.execute("select * from tableNAme")
resultset = cursor.fetchall()
result = convert_to_dict(cursor.description, resultset)
print(result)
resultset = cursor.fetchone()
result = convert_to_dict(cursor.description, resultset)
print(result)
Contents of './config.py'
#!/usr/bin/python
PGCONF = {
"user": "postgres",
"password": "postgres",
"host": "localhost",
"database": "database_name"
}
contents of './main.py'
#!/usr/bin/python
from config import PGCONF
import psycopg2
import psycopg2.extras
# open connection
conn = psycopg2.connect(**PGCONF)
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
# declare lambda function
fetch_all_as_dict = lambda cursor: [dict(row) for row in cursor]
# execute any query of your choice
cur.execute("""select * from table_name limit 1""")
# get all rows as list of dicts
print(fetch_all_as_dict(cur))
# close cursor and connection
cur.close()
conn.close()
Python (2.7.3) is violating my mysql-connector cursor in some strange way when I return it from a function. This first example works fine...
cnx = connect()
sql = "SELECT * FROM MyTable"
cursor = cnx.cursor()
cursor.execute(sql)
row = cursor.fetchone()
However, if I return the cursor and attempt the fetchone() (or a fetchall()) from outside, it throws an exception...
def run_query():
cnx = connect()
sql = "SELECT * FROM MyTable"
cursor = cnx.cursor()
cursor.execute(sql)
return cursor
mycursor = run_query()
row = mycursor.fetchone()
It throws...
File "/usr/lib/pymodules/python2.7/mysql/connector/cursor.py", line 533, in fetchone
row = self._fetch_row()
File "/usr/lib/pymodules/python2.7/mysql/connector/cursor.py", line 508, in _fetch_row
(row, eof) = self.db().protocol.get_row()
AttributeError: 'NoneType' object has no attribute 'protocol'
This is in spite of the fact that "print type(mycursor)" will print "mysql.connector.cursor.MySQLCursor"
What type of unholy molestation is Python performing on objects returned from functions? (Keep in mind that it will do this to cursors passed within a module... so, it's not like the object passed out of the "import mysql.connector" scope... )
I do not have MySQL immediately available, but as Preet Sangha mentioned, when you connect to the database inside the function and return the cursor, your cnx variable goes out of scope when the function exits, so the database connection closes and your cursor references a closed database connection.
This is not the case in your top code example, which may explain why it works and why the bottom example does not.
Can you print type(connect) in your function?
Sample:
>>> import MySQLdb as mydb
>>> def runQuery(sql):
... db = mydb.connect('localhost', 'testuser', 'test', 'test')
... cur = db.cursor()
... cur.execute(sql)
... data = cur.fetchall()
... print "Query :: %s" %sql
... print "Result:: %s" %data
... return cur
...
>>>
>>> cursor = runQuery("SELECT VERSION()")
Query :: SELECT VERSION()
Result:: ('5.6.11-log',)
>>>
>>> cursor.execute("SELECT * FROM EMPLOYEES")
3L
>>> data = cursor.fetchall()
>>>
>>> print data
(('JOHN', 30L, 23000.0), ('SONY', 26L, 14000.0), ('SMITH', 53L, 123000.0))
>>>
>>>
I would like to get the result of the fetchall operation in a list instead of tuple of tuple or tuple of dictionaries.
For example,
cursor = connection.cursor() #Cursor could be a normal cursor or dict cursor
query = "Select id from bs"
cursor.execute(query)
row = cursor.fetchall()
Now, the problem is the resultant row is either ((123,),(234,)) or ({'id':123}, {'id':234})
What I am looking for is (123,234) or [123,234]. Be best if I can save on parsing the resulset.
And what about list comprehensions? If result is ((123,), (234,), (345,)):
>>> row = [item[0] for item in cursor.fetchall()]
>>> row
[123, 234, 345]
If result is ({'id': 123}, {'id': 234}, {'id': 345}):
>>> row = [item['id'] for item in cursor.fetchall()]
>>> row
[123, 234, 345]
I'm sure that after all this time, you've solved this problem, however, for some people who may not know how to get the values of a cursor as a dictionary using MySQLdb, you can use this method found here:
import MySQLdb as mdb
con = mdb.connect('localhost', 'testuser', 'test623', 'testdb')
with con:
cur = con.cursor(mdb.cursors.DictCursor)
cur.execute("SELECT * FROM Writers LIMIT 4")
rows = cur.fetchall()
for row in rows:
print row["Id"], row["Name"]
This old Q comes up on Google while searching for flattening db queries, so here are more suggestions...
Consider a fast list-flattening iterator.
Others answers use fetchall() which first loads all rows in memory, then iterates over that to make a new list. Could be inefficient. Could combine with MySQL so-called server side cursor:
# assume mysql on localhost with db test and table bs
import itertools
import MySQLdb
import MySQLdb.cursors
conn = MySQLdb.connect(host='localhost',db='test',
cursorclass=MySQLdb.cursors.SSCursor )
cursor = conn.cursor()
# insert a bunch of rows
cursor.executemany('INSERT INTO bs (id) VALUES (%s)',zip(range(1,10000)) )
conn.commit()
# retrieve and listify
cursor.execute("select id from bs")
list_of_ids = list(itertools.chain.from_iterable(cursor))
len(list_of_ids)
#9999
conn.close()
But the question is also tagged Django, which has a nice single field query flattener
class Bs(models.Model):
id_field = models.IntegerField()
list_of_ids = Bs.objects.values_list('id_field', flat=True)
Make your cursor object in this manner:
db = MySQLdb.connect("IP", "user", "password", "dbname")
cursor = db.cursor(MySQLdb.cursors.DictCursor)
Then when you perform cursor.fetchall() on a query, a tuple of dictionaries will be obtained, which you can later convert to a list.
data = cursor.fetchall()
data = list(data)
list= [list[0] for list in cursor.fetchall()]
this will render results in one list like - list = [122,45,55,44...]
If there is only one field, i can use this to make a list from database:
def getFieldAsList():
kursor.execute("Select id from bs")
id_data = kursor.fetchall()
id_list = []
for index in range(len(id_data)):
id_list.append(id_data[index][0])
return id_list
cursor.execute("""Select * From bs WHERE (id = %s)""",(id))
cursor.fetchall()
I would like to fetch an array in MySQL. Can someone please tell me how to use Python using MySQLdb to do so?
For example, this is what I would like to do in Python:
<?php
require_once('Config.php');
$q = mysql_query("SELECT * FROM users WHERE firstname = 'namehere'");
$data = mysql_fetch_array($q);
echo $data['lastname'];
?>
Thanks.
In python you have dictionary=True, I have tested in python3. This returns directory which is much similar to associative array in php.
eg.
import mysql.connector
cnx = mysql.connector.connect(user='root', password='',host='127.0.0.1',database='test1')
cursor = cnx.cursor(dictionary=True)
sql= ("SELECT * FROM `users` WHERE id>0")
cursor.execute(sql)
results = cursor.fetchall()
print(results)
You can use this (dictionary=True):
import mysql.connector
db = mysql.connector.connect(user='root', password='',host='127.0.0.1', database='test1')
cursor = db.cursor(dictionary=True)
cursor.execute("SELECT * FROM table")
for row in cursor:
print(row['column'])
Install MySQLdb (the drivers for MySQL for Python). Type pip install mysql-python
Read up on the Python DB API, which is the standard way to access databases in Python.
Then, try this:
>>> import MySQLdb
>>> connection = MySQLdb.connect(database='test')
>>> cursor = connection.cursor()
>>> cursor.execute('SELECT * FROM users WHERE firstname = %s',('somename',))
>>> results = cursor.fetchall()
>>> for i in results:
print i
I would use SQLAlchemy. Something like this would do the trick:
engine = create_engine('mysql://username:password#host:port/database')
connection = engine.connect()
result = connection.execute("select username from users")
for row in result:
print "username:", row['username']
connection.close()
Try:
import MySQLdb
connection = MySQLdb.connect(host="localhost", # your host
user="root", # username
passwd="password", # password
db="frateData") # name of the database)
cursor = connection.cursor(MySQLdb.cursors.DictCursor)
cursor.execute('SELECT * FROM users WHERE firstname = %s',['namehere'])
data = cursor.fetchall()
print data['lastname']
Please note that by initiating your cursor by passing the following parameter: "MySQLdb.cursors.DictCursor"
a list instead of an array is returned so you can reference the data with their key name, which in your case in lastname.