We Keep Coding

Duplicated values as an output of python function

I would like create a Python function for multiple SQL inserts. This function I will use in my Airflow DAG for inserts into Snowflake db. I will need do create an SnowflakeOperator which will use this function. I'm just start to use Airflow so please correct me if I'm wrong.
My example:
I'm connecting to Snowflake db in order to get data from table with information schema name and table name. This output I'm using for inserts per schema. I'm selecting schema and creating variable my_schema = 'my_schema'.
First approach:
sql = "SELECT SCHEMA, TABLE FROM TABLE"
cur.execute(sql)
df = pd.DataFrame.from_records(iter(cur), columns=[x[0] for x in cur.description])
my_dict = dict()
for i in df['SCHEMA'].unique().tolist():
df_x = df[df['SCHEMA'] == i]
my_dict[i] = df_x['TABLE'].tolist()
for schema, tables in my_dict.items():
for table in tables:
query = f"INSERT INTO {schema}.{table} SELECT * FROM {schema}.{table} where col2 = 1;"
try:
cur.execute(query)
except snowflake.connector.errors.ProgrammingError as e:
# Something went wrong with the insert
logging.error(f"Inserting in {schema}.{table}: {e}")
conn.close()
For testing I created pandas datframe with two columns schema and table.
data = [['test', 'table01'], ['test', 'table02'], ['my_schema', 'table03'], ['schemaxxx', 'table04']]
# Create the pandas DataFrame
df_new = pd.DataFrame(data, columns=['schema', 'table'])
I created function for inserts.
my_schema = 'my_schema'
def my_insert_fnc(df):
my_dict = dict()
for i in df['schema'].unique().tolist():
df_x = df[df['schema'] == i]
my_dict[i] = df_x['table'].tolist()
sql_list = []
for schema, tables in my_dict.items():
for table in tables:
if schema == my_schema:
sql_list.append(f"INSERT INTO {schema}.{table} SELECT * FROM {schema}.{table} where col2 = 1;")
print(sql_list)
But I'm getting duplicates.
my_insert_fnc(df_new)
['INSERT INTO my_schema.table03 SELECT * FROM my_schema.table03 where col2 = 1;']
['INSERT INTO my_schema.table03 SELECT * FROM my_schema.table03 where col2 = 1;']
I would like to remove duplicates and logging errors.
try:
cur.execute(query)
except snowflake.connector.errors.ProgrammingError as e:
# Something went wrong with the insert
logging.error(f"Inserting in {schema}.{table}: {e}")
This function as I mentioned I need to use in my Airflow DAG so It needs to give me a string output in order to use it in SnowflakeOperator. Please correct me if I'm wrong.

Django MySQL raw query does not return results

I'm trying to run a raw query in Django. I am not allowed to use ORM.
I use Django MySQL backend.
If I do basic queries, without parametrizing, the database returns results without problems.
The query I want to run (not returning any results):
from django.db import connection
def get_data(variant):
results = []
cursor = connection.cursor()
cursor.execute("SELECT b.spec_id, b.h_loss, c.gen_type FROM `db-dummy`.spec_gen_data c JOIN `db-dummy`.gen_info a ON a.record_id = c.gen_id JOIN `db-dummy`.spec_data b ON b.record_id = c.spec_id WHERE b.h_loss = 1 AND (a.ref_gen = %s OR a.detail_ref_gen = %s) AND c.gen_type BETWEEN 1 AND 5 ORDER BY a.gen_name;", ('{}%'.format(variant),'{}%'.format(variant),))
columns = [column[0] for column in cursor.description]
results = []
for row in cursor.fetchall():
results.append(dict(zip(columns, row)))
return results
Is there something wrong with the syntax?
I am not getting any error, just results = [] after executing the query and I am sure that that query should return results.

Use LIKE instead of = when comparing Strings with wild cards
cursor.execute("SELECT b.spec_id, b.h_loss, c.gen_type FROM `db-dummy`.spec_gen_data c JOIN `db-dummy`.gen_info a ON a.record_id = c.gen_id JOIN `db-dummy`.spec_data b ON b.record_id = c.spec_id WHERE b.h_loss = 1 AND (a.ref_gen LIKE %s OR a.detail_ref_gen LIKE %s) AND c.gen_type BETWEEN 1 AND 5 ORDER BY a.gen_name;", ('{}%'.format(variant),'{}%'.format(variant),))

query from postgresql using python as dictionary

I'm using Python 2.7 and postgresql 9.1.
Trying to get dictionary from query, I've tried the code as described here:
http://wiki.postgresql.org/wiki/Using_psycopg2_with_PostgreSQL
import psycopg2
import psycopg2.extras
conn = psycopg2.connect("dbname=mydb host=localhost user=user password=password")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute ("select * from port")
type(cur.fetchall())
It is printing the next answer:
<type 'list'>
printing the item itself, show me that it is list.
The excepted answer was dictionary.
Edit:
Trying the next:
ans = cur.fetchall()[0]
print ans
print type(ans)
returns
[288, 'T', 51, 1, 1, '192.168.39.188']
<type 'list'>

Tnx a lot Andrey Shokhin ,
full answer is:
#!/var/bin/python
import psycopg2
import psycopg2.extras
conn = psycopg2.connect("dbname=uniart4_pr host=localhost user=user password=password")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute ("select * from port")
ans =cur.fetchall()
ans1 = []
for row in ans:
ans1.append(dict(row))
print ans1 #actually it's return

It's normal: when you call .fetchall() method returns list of tuples. But if you write
type(cur.fetchone())
it will return only one tuple with type:
<class 'psycopg2.extras.DictRow'>
After this you can use it as list or like dictionary:
cur.execute('SELECT id, msg FROM table;')
rec = cur.fetchone()
print rec[0], rec['msg']
You can also use a simple cursor iterator:
res = [json.dumps(dict(record)) for record in cursor] # it calls .fetchone() in loop

Perhaps to optimize it further we can have
#!/var/bin/python
import psycopg2
import psycopg2.extras
def get_dict_resultset(sql):
conn = psycopg2.connect("dbname=pem host=localhost user=postgres password=Drupal#1008")
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cur.execute (sql)
ans =cur.fetchall()
dict_result = []
for row in ans:
dict_result.append(dict(row))
return dict_result
sql = """select * from tablename"""
return get_dict_resultset(sql)

If you don't want to use a psycopg2.extras.DictCursor you can create a list of dictionaries for the results using cursor.description:
# connect
connection = psycopg2.connect()
cursor = connection.cursor()
# query
cursor.execute("SELECT * FROM myTable")
# transform result
columns = list(cursor.description)
result = cursor.fetchall()
# make dict
results = []
for row in result:
row_dict = {}
for i, col in enumerate(columns):
row_dict[col.name] = row[i]
results.append(row_dict)
# display
print(result)
I use the following function fairly regularly:
def select_query_dict(connection, query, data=[]):
"""
Run generic select query on db, returns a list of dictionaries
"""
logger.debug('Running query: {}'.format(query))
# Open a cursor to perform database operations
cursor = connection.cursor()
logging.debug('Db connection succesful')
# execute the query
try:
logger.info('Running query.')
if len(data):
cursor.execute(query, data)
else:
cursor.execute(query)
columns = list(cursor.description)
result = cursor.fetchall()
logging.debug('Query executed succesfully')
except (Exception, psycopg2.DatabaseError) as e:
logging.error(e)
cursor.close()
exit(1)
cursor.close()
# make dict
results = []
for row in result:
row_dict = {}
for i, col in enumerate(columns):
row_dict[col.name] = row[i]
results.append(row_dict)
return results

In addition to just return only the query results as a list of dictionaries, I would suggest returning key-value pairs (column-name:row-value). Here my suggestion:
import psycopg2
import psycopg2.extras
conn = None
try:
conn = psycopg2.connect("dbname=uniart4_pr host=localhost user=user password=password")
with conn.cursor(cursor_factory=psycopg2.extras.DictCursor) as cursor:
cursor.execute("SELECT * FROM table")
column_names = [desc[0] for desc in cursor.description]
res = cursor.fetchall()
cursor.close()
return map(lambda x: dict(zip(column_names, x)), res))
except (Exception, psycopg2.DatabaseError) as e:
logger.error(e)
finally:
if conn is not None:
conn.close()

There is a built in solution to get your result as a collection of dictionary:
from psycopg2.extras import RealDictCursor
cur = conn.cursor(cursor_factory=RealDictCursor)
Modified from: https://www.peterbe.com/plog/from-postgres-to-json-strings, copyright 2013 Peter Bengtsson

For me when I convert the row to dictionary failed (solutions mentioned by others)and also could not use cursor factory.
I am using PostgreSQL 9.6.10, Below code worked for me but I am not sure if its the right way to do it.
def convert_to_dict(columns, results):
"""
This method converts the resultset from postgres to dictionary
interates the data and maps the columns to the values in result set and converts to dictionary
:param columns: List - column names return when query is executed
:param results: List / Tupple - result set from when query is executed
:return: list of dictionary- mapped with table column name and to its values
"""
allResults = []
columns = [col.name for col in columns]
if type(results) is list:
for value in results:
allResults.append(dict(zip(columns, value)))
return allResults
elif type(results) is tuple:
allResults.append(dict(zip(columns, results)))
return allResults
Way to use it:
conn = psycopg2.connect("dbname=pem host=localhost user=postgres,password=Drupal#1008")
cur = conn.cursor()
cur.execute("select * from tableNAme")
resultset = cursor.fetchall()
result = convert_to_dict(cursor.description, resultset)
print(result)
resultset = cursor.fetchone()
result = convert_to_dict(cursor.description, resultset)
print(result)

Contents of './config.py'
#!/usr/bin/python
PGCONF = {
"user": "postgres",
"password": "postgres",
"host": "localhost",
"database": "database_name"
}
contents of './main.py'
#!/usr/bin/python
from config import PGCONF
import psycopg2
import psycopg2.extras
# open connection
conn = psycopg2.connect(**PGCONF)
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
# declare lambda function
fetch_all_as_dict = lambda cursor: [dict(row) for row in cursor]
# execute any query of your choice
cur.execute("""select * from table_name limit 1""")
# get all rows as list of dicts
print(fetch_all_as_dict(cur))
# close cursor and connection
cur.close()
conn.close()

Python MySQLDB: Get the result of fetchall in a list

I would like to get the result of the fetchall operation in a list instead of tuple of tuple or tuple of dictionaries.
For example,
cursor = connection.cursor() #Cursor could be a normal cursor or dict cursor
query = "Select id from bs"
cursor.execute(query)
row = cursor.fetchall()
Now, the problem is the resultant row is either ((123,),(234,)) or ({'id':123}, {'id':234})
What I am looking for is (123,234) or [123,234]. Be best if I can save on parsing the resulset.

And what about list comprehensions? If result is ((123,), (234,), (345,)):
>>> row = [item[0] for item in cursor.fetchall()]
>>> row
[123, 234, 345]
If result is ({'id': 123}, {'id': 234}, {'id': 345}):
>>> row = [item['id'] for item in cursor.fetchall()]
>>> row
[123, 234, 345]

I'm sure that after all this time, you've solved this problem, however, for some people who may not know how to get the values of a cursor as a dictionary using MySQLdb, you can use this method found here:
import MySQLdb as mdb
con = mdb.connect('localhost', 'testuser', 'test623', 'testdb')
with con:
cur = con.cursor(mdb.cursors.DictCursor)
cur.execute("SELECT * FROM Writers LIMIT 4")
rows = cur.fetchall()
for row in rows:
print row["Id"], row["Name"]

This old Q comes up on Google while searching for flattening db queries, so here are more suggestions...
Consider a fast list-flattening iterator.
Others answers use fetchall() which first loads all rows in memory, then iterates over that to make a new list. Could be inefficient. Could combine with MySQL so-called server side cursor:
# assume mysql on localhost with db test and table bs
import itertools
import MySQLdb
import MySQLdb.cursors
conn = MySQLdb.connect(host='localhost',db='test',
cursorclass=MySQLdb.cursors.SSCursor )
cursor = conn.cursor()
# insert a bunch of rows
cursor.executemany('INSERT INTO bs (id) VALUES (%s)',zip(range(1,10000)) )
conn.commit()
# retrieve and listify
cursor.execute("select id from bs")
list_of_ids = list(itertools.chain.from_iterable(cursor))
len(list_of_ids)
#9999
conn.close()
But the question is also tagged Django, which has a nice single field query flattener
class Bs(models.Model):
id_field = models.IntegerField()
list_of_ids = Bs.objects.values_list('id_field', flat=True)

Make your cursor object in this manner:
db = MySQLdb.connect("IP", "user", "password", "dbname")
cursor = db.cursor(MySQLdb.cursors.DictCursor)
Then when you perform cursor.fetchall() on a query, a tuple of dictionaries will be obtained, which you can later convert to a list.
data = cursor.fetchall()
data = list(data)

list= [list[0] for list in cursor.fetchall()]
this will render results in one list like - list = [122,45,55,44...]

If there is only one field, i can use this to make a list from database:
def getFieldAsList():
kursor.execute("Select id from bs")
id_data = kursor.fetchall()
id_list = []
for index in range(len(id_data)):
id_list.append(id_data[index][0])
return id_list

cursor.execute("""Select * From bs WHERE (id = %s)""",(id))
cursor.fetchall()

MySQL: Get column name or alias from query

I'm not asking for the SHOW COLUMNS command.
I want to create an application that works similarly to heidisql, where you can specify an SQL query and when executed, returns a result set with rows and columns representing your query result. The column names in the result set should match your selected columns as defined in your SQL query.
In my Python program (using MySQLdb) my query returns only the row and column results, but not the column names. In the following example the column names would be ext, totalsize, and filecount. The SQL would eventually be external from the program.
The only way I can figure to make this work, is to write my own SQL parser logic to extract the selected column names.
Is there an easy way to get the column names for the provided SQL?
Next I'll need to know how many columns does the query return?
# Python
import MySQLdb
#===================================================================
# connect to mysql
#===================================================================
try:
db = MySQLdb.connect(host="myhost", user="myuser", passwd="mypass",db="mydb")
except MySQLdb.Error, e:
print "Error %d: %s" % (e.args[0], e.args[1])
sys.exit (1)
#===================================================================
# query select from table
#===================================================================
cursor = db.cursor ()
cursor.execute ("""\
select ext,
sum(size) as totalsize,
count(*) as filecount
from fileindex
group by ext
order by totalsize desc;
""")
while (1):
row = cursor.fetchone ()
if row == None:
break
print "%s %s %s\n" % (row[0], row[1], row[2])
cursor.close()
db.close()

cursor.description will give you a tuple of tuples where [0] for each is the column header.
num_fields = len(cursor.description)
field_names = [i[0] for i in cursor.description]

This is the same as thefreeman but more in pythonic way using list and dictionary comprehension
columns = cursor.description
result = [{columns[index][0]:column for index, column in enumerate(value)} for value in cursor.fetchall()]
pprint.pprint(result)

Similar to #James answer, a more pythonic way can be:
fields = [field_md[0] for field_md in cursor.description]
result = [dict(zip(fields,row)) for row in cursor.fetchall()]
You can get a single column with list comprehension over the result:
extensions = [row['ext'] for row in result)
or filter results using an additional if in the list comprehension:
large = [row for row in result if row['filesize'] > 1024 and row['filesize'] < 4096]
or accumulate values for filtered columns:
totalTxtSize = reduce(
lambda x,y: x+y,
filter(lambda x: x['ext'].lower() == 'txt', result)
)

I think this should do what you need (builds on the answer above) . I am sure theres a more pythony way to write it, but you should get the general idea.
cursor.execute(query)
columns = cursor.description
result = []
for value in cursor.fetchall():
tmp = {}
for (index,column) in enumerate(value):
tmp[columns[index][0]] = column
result.append(tmp)
pprint.pprint(result)

You could also use MySQLdb.cursors.DictCursor. This turns your result set into a python list of python dictionaries, although it uses a special cursor, thus technically less portable than the accepted answer. Not sure about speed. Here's the edited original code that uses this.
#!/usr/bin/python -u
import MySQLdb
import MySQLdb.cursors
#===================================================================
# connect to mysql
#===================================================================
try:
db = MySQLdb.connect(host='myhost', user='myuser', passwd='mypass', db='mydb', cursorclass=MySQLdb.cursors.DictCursor)
except MySQLdb.Error, e:
print 'Error %d: %s' % (e.args[0], e.args[1])
sys.exit(1)
#===================================================================
# query select from table
#===================================================================
cursor = db.cursor()
sql = 'SELECT ext, SUM(size) AS totalsize, COUNT(*) AS filecount FROM fileindex GROUP BY ext ORDER BY totalsize DESC;'
cursor.execute(sql)
all_rows = cursor.fetchall()
print len(all_rows) # How many rows are returned.
for row in all_rows: # While loops always make me shudder!
print '%s %s %s\n' % (row['ext'], row['totalsize'], row['filecount'])
cursor.close()
db.close()
Standard dictionary functions apply, for example, len(row[0]) to count the number of columns for the first row, list(row[0]) for a list of column names (for the first row), etc. Hope this helps!

This is only an add-on to the accepted answer:
def get_results(db_cursor):
desc = [d[0] for d in db_cursor.description]
results = [dotdict(dict(zip(desc, res))) for res in db_cursor.fetchall()]
return results
where dotdict is:
class dotdict(dict):
__getattr__ = dict.get
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
This will allow you to access much easier the values by column names.
Suppose you have a user table with columns name and email:
cursor.execute('select * from users')
results = get_results(cursor)
for res in results:
print(res.name, res.email)

Something similar to the proposed solutions, only the result is json with column_header : vaule for db_query ie sql.
cur = conn.cursor()
cur.execute(sql)
res = [dict((cur.description[i][0], value) for i, value in enumerate(row)) for row in cur.fetchall()]
output json example:
[
{
"FIRST_ROW":"Test 11",
"SECOND_ROW":"Test 12",
"THIRD_ROW":"Test 13"
},
{
"FIRST_ROW":"Test 21",
"SECOND_ROW":"Test 22",
"THIRD_ROW":"Test 23"
}
]

Looks like MySQLdb doesn't actually provide a translation for that API call. The relevant C API call is mysql_fetch_fields, and there is no MySQLdb translation for that

Try:
cursor.column_names
mysql connector version:
mysql.connector.__version__
'2.2.9'

You can also do this to just get the field titles:
table = cursor.description
check = 0
for fields in table:
for name in fields:
if check < 1:
print(name),
check +=1
check =0

cursor.column_names is a nice and simple one.

column_names = cursor.field_names

found an easy way of having colums like sql using pymysql and pandas
import pymysql
import pandas as pd
db = pymysql.connect(host="myhost", user="myuser", passwd="mypass", db="mydb")
query = """SELECT ext
SUM(size) as totalsize,
COUNT(*) as filecount
FROM fileindex
GROUP BY ext
ORDER BY totalsize DESC;
"""
df = pd.read_sql_query(query,db)
the DataFrame will have column names ext,totalsize,filecount by default no need to do additional stuff.
for example in my case: