I am writing a script to query mariadb database which returns query output like this
('101', datetime.datetime(2023, 1, 27, 14, 8, 39), datetime.datetime(2023, 1, 27, 14, 8, 39)
instead I want output like this
('101', '2023-01-27 14:08:39.399', 0, '2023-01-27 14:08:39.399')
try:
conn = mariadb.connect()
except mariadb.Error as e:
print(f"Error connecting to MariaDB Platform: {e}")
sys.exit(1)
# Get Cursor
cur = conn.cursor()
cur.execute("SELECT * FROM <table>")
myresult = cur.fetchall()
print(myresult)
cur.close()
I had gotten similar issue when trying with pyodbc so I used my custom output converter but I didn't find any for this library
here's how i did it in pyodbc
`
# function to convert datetime.date(2022, 10, 12) with data type date to string
def convert_date_to_string(raw_bytes):
tup = struct.unpack('3h', raw_bytes)
return datetime.date(tup[0], tup[1], tup[2]).strftime('%Y-%m-%d')
conn.add_output_converter(po.SQL_TYPE_DATE, convert_date_to_string)
Kindly suggest a similar approach for maraidb connector as I don't want add parser for every query I make instead I want to do something global like one mentioned aboved as in pyodbc.
It is the default behavior in database driver that a database value will be converted to the corresponding python data type.
If you need to convert it into another value, you can either do that on SQL level by using CAST function
>>> cursor.execute("select cast(now() as char)")
>>> cursor.fetchone()
('2023-02-19 14:42:51',)
or by using a converter:
>>> import mariadb
>>> from mariadb.constants import FIELD_TYPE
>>>
>>> def date_to_str(date):
... return str(date)
...
>>> conversions= {
... **{FIELD_TYPE.DATETIME: date_to_str}
... }
>>>
>>> conn= mariadb.connect(converter=conversions)
>>> cursor=conn.cursor()
>>> cursor.execute("SELECT NOW()")
>>> cursor.fetchone()
('2023-02-19 14:50:29',)
Related
I follow this link to query Azure database.
import pyodbc
server = 'your_server.database.windows.net'
database = 'your_database'
username = 'your_username'
password = 'your_password'
driver= '{ODBC Driver 13 for SQL Server}'
cnxn = pyodbc.connect('DRIVER='+driver+';PORT=1433;SERVER='+server+';PORT=1443;DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
cursor.execute("SELECT * FROM FinancialRecord where deleted=0")
row = cursor.fetchone()
while row:
print (str(row[0]) + " " + str(row[1]))
row = cursor.fetchone()
When I run the code above, it show the error.
Traceback (most recent call last):
File "sqltest.py", line 10, in
row = cursor.fetchone()
pyodbc.ProgrammingError: ('ODBC SQL type -155 is not yet supported. column-index=2 type=-155', 'HY106')
I am new to Azure. Anyone can help?
pyodbc supports Output Converter functions that we can use when a database returns an SQL type that pyodbc does not support natively. The example on the Wiki page linked above will perform a client-side conversion similar to what would be achieved by a CAST to [N]VARCHAR on the server:
import struct
import pyodbc
conn = pyodbc.connect("DSN=myDb")
def handle_datetimeoffset(dto_value):
# ref: https://github.com/mkleehammer/pyodbc/issues/134#issuecomment-281739794
tup = struct.unpack("<6hI2h", dto_value) # e.g., (2017, 3, 16, 10, 35, 18, 0, -6, 0)
tweaked = [tup[i] // 100 if i == 6 else tup[i] for i in range(len(tup))]
return "{:04d}-{:02d}-{:02d} {:02d}:{:02d}:{:02d}.{:07d} {:+03d}:{:02d}".format(*tweaked)
crsr = conn.cursor()
# create test data
crsr.execute("CREATE TABLE #dto_test (id INT PRIMARY KEY, dto_col DATETIMEOFFSET)")
crsr.execute("INSERT INTO #dto_test (id, dto_col) VALUES (1, '2017-03-16 10:35:18 -06:00')")
conn.add_output_converter(-155, handle_datetimeoffset)
value = crsr.execute("SELECT dto_col FROM #dto_test WHERE id=1").fetchval()
print(value)
crsr.close()
conn.close()
which prints
2017-03-16 10:35:18.0000000 -06:00
ODBC SQL type -155 corresponds to the SQL Server type DatetimeOFFSET and the ODBC type SQL_SS_TIMESTAMPOFFSET. The mapping between ODBC types and SQL Server types is described in this documentation page. The error message says that this SQL Server datatype is currently unsupported by the Python ODBC API.
To work around the issue, you will need to change your query to avoid querying columns with DatetimeOFFSET datatype. One way to proceed is to identify the columns in your FinancialRecord table that have a DatetimeOFFSET datatype and convert them to a nvarchar(100) type.
SELECT CAST(MyColumn as AS nvarchar(100)) AS MyColumnAsNVarChar', ...
FROM FinancialRecord where deleted=0
I follow this link to query Azure database.
import pyodbc
server = 'your_server.database.windows.net'
database = 'your_database'
username = 'your_username'
password = 'your_password'
driver= '{ODBC Driver 13 for SQL Server}'
cnxn = pyodbc.connect('DRIVER='+driver+';PORT=1433;SERVER='+server+';PORT=1443;DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
cursor.execute("SELECT * FROM FinancialRecord where deleted=0")
row = cursor.fetchone()
while row:
print (str(row[0]) + " " + str(row[1]))
row = cursor.fetchone()
When I run the code above, it show the error.
Traceback (most recent call last):
File "sqltest.py", line 10, in
row = cursor.fetchone()
pyodbc.ProgrammingError: ('ODBC SQL type -155 is not yet supported. column-index=2 type=-155', 'HY106')
I am new to Azure. Anyone can help?
pyodbc supports Output Converter functions that we can use when a database returns an SQL type that pyodbc does not support natively. The example on the Wiki page linked above will perform a client-side conversion similar to what would be achieved by a CAST to [N]VARCHAR on the server:
import struct
import pyodbc
conn = pyodbc.connect("DSN=myDb")
def handle_datetimeoffset(dto_value):
# ref: https://github.com/mkleehammer/pyodbc/issues/134#issuecomment-281739794
tup = struct.unpack("<6hI2h", dto_value) # e.g., (2017, 3, 16, 10, 35, 18, 0, -6, 0)
tweaked = [tup[i] // 100 if i == 6 else tup[i] for i in range(len(tup))]
return "{:04d}-{:02d}-{:02d} {:02d}:{:02d}:{:02d}.{:07d} {:+03d}:{:02d}".format(*tweaked)
crsr = conn.cursor()
# create test data
crsr.execute("CREATE TABLE #dto_test (id INT PRIMARY KEY, dto_col DATETIMEOFFSET)")
crsr.execute("INSERT INTO #dto_test (id, dto_col) VALUES (1, '2017-03-16 10:35:18 -06:00')")
conn.add_output_converter(-155, handle_datetimeoffset)
value = crsr.execute("SELECT dto_col FROM #dto_test WHERE id=1").fetchval()
print(value)
crsr.close()
conn.close()
which prints
2017-03-16 10:35:18.0000000 -06:00
ODBC SQL type -155 corresponds to the SQL Server type DatetimeOFFSET and the ODBC type SQL_SS_TIMESTAMPOFFSET. The mapping between ODBC types and SQL Server types is described in this documentation page. The error message says that this SQL Server datatype is currently unsupported by the Python ODBC API.
To work around the issue, you will need to change your query to avoid querying columns with DatetimeOFFSET datatype. One way to proceed is to identify the columns in your FinancialRecord table that have a DatetimeOFFSET datatype and convert them to a nvarchar(100) type.
SELECT CAST(MyColumn as AS nvarchar(100)) AS MyColumnAsNVarChar', ...
FROM FinancialRecord where deleted=0
How to convert
cursor.execute("SELECT strftime('%m.%d.%Y %H:%M:%S', timestamp, 'localtime'), temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
for row in results[:-1]:
row=results[-1]
rowstr="['{0}',{1}]\n".format(str(row[0]),str(row[1]))
temp_chart_table+=rowstr
result
['01.15.2015 21:38:52',21.812]
into dictionary output in form of:
[{timestamp:'01.15.2015 21:38:52',temp:21.812}]
Edit
This is fetchone sample I currenyly use and it works fine:
def get_avg():
conn=sqlite3.connect(dbname)
curs=conn.cursor()
curs.execute("SELECT ROUND(avg(temp), 2.2) FROM data WHERE timestamp>datetime('now','-1 hour') AND timestamp<=datetime('now')")
rowavg=curs.fetchone()
#print rowavg
#rowstrmin=format(str(rowavg[0]))
#return rowstrmin
**d = [{"avg":rowavg[0]}]**
return d
conn.close()
#print get_avg()
schema = {"avg": ("number", "avg")}
data = get_avg()
# Loading it into gviz_api.DataTable
data_table = gviz_api.DataTable(schema)
data_table.LoadData(data)
json = data_table.ToJSon()
#print results
#print "Content-type: application/json\n\n"
print "Content-type: application/json"
print
print json
Then I make jQuery call and pass it into javascript and found help for that in here
ajax json query directly to python generated html gets undefined
As I can see you are using format to write in the form of a string.
Note from the docs
it is not possible to use { and } as fill char while using the str.format() method
To make it look like a dictionary you can do
"[{timestamp:'%s',temp:%s}]\n"%(str(row[0]),str(row[1]))
But if you want to make it a dictionary then you will have to do
row_dic = [{'timestamp':row[0],'temp':row[1]}]
Try this instead:
cursor.execute("SELECT strftime('%m.%d.%Y %H:%M:%S', timestamp, 'localtime'), temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
temp_chart_table = []
for row in results:
temp_chart_table.append({'timestamp': row[0], 'temp': row[1]})
In most of the python database adapters you can use a DictCursor to retrieve records using an interface similar to the Python dictionaries instead of the tuples.
Using psycopg2:
>>> dict_cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
>>> dict_cur.execute("INSERT INTO test (num, data) VALUES(%s, %s)",
... (100, "abc'def"))
>>> dict_cur.execute("SELECT * FROM test")
>>> rec = dict_cur.fetchone()
>>> rec['id']
1
>>> rec['num']
100
>>> rec['data']
"abc'def"
Using MySQLdb:
>>> import MySQLdb
>>> import MySQLdb.cursors
>>> myDb = MySQLdb.connect(user='andy47', passwd='password', db='db_name', cursorclass=MySQLdb.cursors.DictCursor)
>>> myCurs = myDb.cursor()
>>> myCurs.execute("SELECT columna, columnb FROM tablea")
>>> firstRow = myCurs.fetchone()
{'columna':'first value', 'columnb':'second value'}
def stuffToDict(stuff):
return {"timestamp":stuff[0],"temp":stuff[1]}
That would be a dictionary. The sample output you showed is a list of dictionaries, which can be achieved by putting square brackets around the dictionary. I don't know why you'd want that, though. Also, because of the missing quotes, it wasn't legal python syntax.
Use MySQLdb's cursor library.
import MySQLdb
import MySQLdb.cursors
conn = MySQLdb.connect(host=db_host, user=db_user, passwd=db_passwd, db=db_schema, port=db_port, cursorclass=MySQLdb.cursors.DictCursor)
cursor = conn.cursor()
cursor.execute("SELECT timestamp, localtime, temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
Then you have access to the results as a dictionary:
>>> results['timestamp']
14146587
>>> results['localtime']
20:08:07
>>> results['temp']
temp_variable_whatever
Python (2.7.3) is violating my mysql-connector cursor in some strange way when I return it from a function. This first example works fine...
cnx = connect()
sql = "SELECT * FROM MyTable"
cursor = cnx.cursor()
cursor.execute(sql)
row = cursor.fetchone()
However, if I return the cursor and attempt the fetchone() (or a fetchall()) from outside, it throws an exception...
def run_query():
cnx = connect()
sql = "SELECT * FROM MyTable"
cursor = cnx.cursor()
cursor.execute(sql)
return cursor
mycursor = run_query()
row = mycursor.fetchone()
It throws...
File "/usr/lib/pymodules/python2.7/mysql/connector/cursor.py", line 533, in fetchone
row = self._fetch_row()
File "/usr/lib/pymodules/python2.7/mysql/connector/cursor.py", line 508, in _fetch_row
(row, eof) = self.db().protocol.get_row()
AttributeError: 'NoneType' object has no attribute 'protocol'
This is in spite of the fact that "print type(mycursor)" will print "mysql.connector.cursor.MySQLCursor"
What type of unholy molestation is Python performing on objects returned from functions? (Keep in mind that it will do this to cursors passed within a module... so, it's not like the object passed out of the "import mysql.connector" scope... )
I do not have MySQL immediately available, but as Preet Sangha mentioned, when you connect to the database inside the function and return the cursor, your cnx variable goes out of scope when the function exits, so the database connection closes and your cursor references a closed database connection.
This is not the case in your top code example, which may explain why it works and why the bottom example does not.
Can you print type(connect) in your function?
Sample:
>>> import MySQLdb as mydb
>>> def runQuery(sql):
... db = mydb.connect('localhost', 'testuser', 'test', 'test')
... cur = db.cursor()
... cur.execute(sql)
... data = cur.fetchall()
... print "Query :: %s" %sql
... print "Result:: %s" %data
... return cur
...
>>>
>>> cursor = runQuery("SELECT VERSION()")
Query :: SELECT VERSION()
Result:: ('5.6.11-log',)
>>>
>>> cursor.execute("SELECT * FROM EMPLOYEES")
3L
>>> data = cursor.fetchall()
>>>
>>> print data
(('JOHN', 30L, 23000.0), ('SONY', 26L, 14000.0), ('SMITH', 53L, 123000.0))
>>>
>>>
Is there any way to get the results from a fetchall() as a dictionary using pymysql?
PyMySQL includes a DictCursor. It does what I think you want. Here's how to use it:
import pymysql
import pymysql.cursors
connection = pymysql.connect(db="test")
cursor = connection.cursor(pymysql.cursors.DictCursor)
cursor.execute("SELECT ...")
https://github.com/PyMySQL/PyMySQL/blob/master/pymysql/tests/test_DictCursor.py
Use pymysql.cursors.DictCursor, which will return rows represented as dictionaries mapping column names to values.
A few ways to use it...
Create a connection object and have all cursors spawned from it be DictCursors:
>>> import pymysql
>>> connection = pymysql.connect(db='foo', cursorclass=pymysql.cursors.DictCursor)
>>> with connection.cursor() as cursor:
... print cursor
...
<pymysql.cursors.DictCursor object at 0x7f87682fefd0>
>>> with connection.cursor() as cursor:
... cursor.execute("SELECT * FROM bar")
... print cursor.fetchall()
...
2
[{u'col2': 'rty', u'col1': 'qwe'}, {u'col2': 'fgh', u'col1': 'asd'}]
Create a DictCursor from an ordinary connection object:
>>> connection = pymysql.connect(db='foo')
>>> with connection.cursor(pymysql.cursors.DictCursor) as cursor:
... print cursor
...
<pymysql.cursors.DictCursor object at 0x7f876830c050>
Connect and create a DictCursor in one line with with:
>>> from pymysql.cursors import DictCursor
>>> with pymysql.connect(db='foo', cursorclass=DictCursor) as cursor:
... print cursor
...
<pymysql.cursors.DictCursor object at 0x7f8767769490>
Use a DictCursor in the cursor() method.
If you mean that you want to fetch two columns, and return them as a dictionary, you can use this method.
def fetch_as_dict(cursor select_query):
'''Execute a select query and return the outcome as a dict.'''
cursor.execute(select_query)
data = cursor.fetchall()
try:
result = dict(data)
except:
msg = 'SELECT query must have exactly two columns'
raise AssertionError(msg)
return result