I have the following code which is attempting to pull several SQL queries from an Access database
import pyodbc
import datetime
conx = pyodbc.connect("Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:\\Users\\Stuart\\PycharmProjects\\untitled\\Databases\\SandP.accdb;")
cursor=conx.cursor()
query=""" SELECT DISTINCT Date_ FROM Closing_prices
WHERE Date_ >= ? AND Date_ < ?"""
params1 = (datetime.date(2011, 8, 10), datetime.date(2014, 4, 30))
cursor.execute(query, params1)
dates = list()
for date in cursor:
dates.append(date[0])
for date in dates:
params2 = date
cursor = conx.cursor()
query= '''SELECT Volatility,Last_price FROM Volatility v,Closing_prices c WHERE c.Date_= ? and v.Date_=c.Date_'''
cursor.execute(query,params2)
for (vol,price) in cursor:
volatility=float(vol)
closing_price=float(price)
cursor.close()
cursor = conx.cursor()
if (date.weekday()==4):
nextDay=(date + datetime.timedelta(days=3))
else:
nextDay=(date + datetime.timedelta(days=1))
query= '''SELECT Date_,Time_, Close_ FROM Intraday_values WHERE (date = ? and time >= ?) or (date = ? and time <= ?)'''
params3 = (date,datetime.time(15, 30),nextDay,datetime.time(15, 14))
cursor.execute(query,params3)
This last bit is throwing up the following error:
Traceback (most recent call last):
File "C:/Users/Stuart/PycharmProjects/untitled/Apache - Copy.py", line 67, in <module>
cursor.execute(query,params3)
pyodbc.Error: ('07002', '[07002] [Microsoft][ODBC Microsoft Access Driver] Too few parameters. Expected 6. (-3010) (SQLExecDirectW)')
The request is attempting to pull out the Date_, Time_ and Close_ items from the table, for the dates passed to the request as it iterates through the list of dates created previously, along with cut off times of "after 15:30 for "date" and "before 15:14" for "date+1".
Firstly, why would this be expecting 6 parameters when there are only 4 question marks (?) in the SQL request - have I not formed this properly?
Also, I took a stab at the parameter creation for a datetime.time. Is this incorrectly formed also?
I'm a bit out of my depth!
Works after changing the query
to
query= '''SELECT Date_,Time_, Close_
FROM Intraday_values
WHERE (Date_ = ? and Time_ >= ?)
OR (Date_ = ? and Time_ <= ?)'''
Related
I follow this link to query Azure database.
import pyodbc
server = 'your_server.database.windows.net'
database = 'your_database'
username = 'your_username'
password = 'your_password'
driver= '{ODBC Driver 13 for SQL Server}'
cnxn = pyodbc.connect('DRIVER='+driver+';PORT=1433;SERVER='+server+';PORT=1443;DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
cursor.execute("SELECT * FROM FinancialRecord where deleted=0")
row = cursor.fetchone()
while row:
print (str(row[0]) + " " + str(row[1]))
row = cursor.fetchone()
When I run the code above, it show the error.
Traceback (most recent call last):
File "sqltest.py", line 10, in
row = cursor.fetchone()
pyodbc.ProgrammingError: ('ODBC SQL type -155 is not yet supported. column-index=2 type=-155', 'HY106')
I am new to Azure. Anyone can help?
pyodbc supports Output Converter functions that we can use when a database returns an SQL type that pyodbc does not support natively. The example on the Wiki page linked above will perform a client-side conversion similar to what would be achieved by a CAST to [N]VARCHAR on the server:
import struct
import pyodbc
conn = pyodbc.connect("DSN=myDb")
def handle_datetimeoffset(dto_value):
# ref: https://github.com/mkleehammer/pyodbc/issues/134#issuecomment-281739794
tup = struct.unpack("<6hI2h", dto_value) # e.g., (2017, 3, 16, 10, 35, 18, 0, -6, 0)
tweaked = [tup[i] // 100 if i == 6 else tup[i] for i in range(len(tup))]
return "{:04d}-{:02d}-{:02d} {:02d}:{:02d}:{:02d}.{:07d} {:+03d}:{:02d}".format(*tweaked)
crsr = conn.cursor()
# create test data
crsr.execute("CREATE TABLE #dto_test (id INT PRIMARY KEY, dto_col DATETIMEOFFSET)")
crsr.execute("INSERT INTO #dto_test (id, dto_col) VALUES (1, '2017-03-16 10:35:18 -06:00')")
conn.add_output_converter(-155, handle_datetimeoffset)
value = crsr.execute("SELECT dto_col FROM #dto_test WHERE id=1").fetchval()
print(value)
crsr.close()
conn.close()
which prints
2017-03-16 10:35:18.0000000 -06:00
ODBC SQL type -155 corresponds to the SQL Server type DatetimeOFFSET and the ODBC type SQL_SS_TIMESTAMPOFFSET. The mapping between ODBC types and SQL Server types is described in this documentation page. The error message says that this SQL Server datatype is currently unsupported by the Python ODBC API.
To work around the issue, you will need to change your query to avoid querying columns with DatetimeOFFSET datatype. One way to proceed is to identify the columns in your FinancialRecord table that have a DatetimeOFFSET datatype and convert them to a nvarchar(100) type.
SELECT CAST(MyColumn as AS nvarchar(100)) AS MyColumnAsNVarChar', ...
FROM FinancialRecord where deleted=0
I'm trying to insert multiple rows in a database table (SQL server) in python:
def write_to_database(values):
conn = pyodbc.connect("Driver={SQL Server};"
"Server=xxxx.database.windows.net;"
f"Database={database};"
"UID=user;"
"PWD=password;")
cursor = conn.cursor()
cursor.executemany("insert into KeyFigures (DateTime, Parameter, CommulativeTime, Value) values (?,?,?,?)", values)
conn.commit()
return()
date = datetime.datetime.now()
atom = date.strftime("%Y-%M-%d") + " " + date.strftime("%H:%M:%S") + ".4526800"
values = ([datetime.datetime.now().isoformat(), 1, "NULL", 2],[datetime.datetime.now().isoformat(), 1, "NULL", 47],[datetime.datetime.now().isoformat(), 1, "NULL", 78])
write_to_database(values)
I tried multiple formats of datetime, string combinations etc. e.g.:
datetime.datetime.now().isoformat()
atom
"2020-02-23 11:30:53.4526800"
"2020-02-23T11:30:53.4526800"
but i keep recieving the same error:
line 50, in write_to_database
cursor.executemany("insert into KeyFigures (DateTime, Parameter, CommulativeTime, Value) values (?,?,?,?)", values)
pyodbc.DataError: ('22007', '[22007] [Microsoft][ODBC SQL Server Driver][SQL Server]Conversion failed when converting date and/or time from character string. (241) (SQLExecDirectW)')
in SSMS the folowing works:
INSERT INTO KeyFigures (DateTime, Parameter, CommulativeTime, Value) VALUES ('2020-02-23 11:30:53.4526800',2,null,21)
how can I solve this error?
***** edit ****
#Mogo thank you very much. this was already very helpfull, but does not work in my code. I still receive the same error. I also tried to insert a single row and this piece of code works (with execute instead of executemany):
def write_to_database(date,parameter,cummulativeTime,value):
conn = pyodbc.connect("Driver={ODBC Driver 17 for SQL Server};"
"Server=xxxx.database.windows.net;"
f"Database={database};"
"UID=user;"
"PWD=password;")
with conn.cursor() as cursor:
cursor.execute(f"INSERT INTO dbo.KeyFigures (DateTime, Parameter, CommulativeTime, Value) values ('{date}',{parameter},{cummulativeTime},{value})")
conn.commit()
return()
date = datetime.datetime.now()
write_to_database(date, 1, "NULL", 43)
it doesnt work without date between quotes. Is this also the problem with the executemany? when i put the questionmark between qoutes ('?' or '?') gives the error that there are only 3 parameters given instead of 4.
As I mentioned in my comment, if you use a strongly type data type (so don't convert it to a string), python and pyodbc will handle this gracefully. I also, however, recommend updating to the ODBC Driver for SQL Server rather than using the old Native SQL Server Driver. I also put the cursor into a with so that it is close gracefully.
For a table I created with the definition below this worked fine, and inserted 2 rows, with the correct date and time values:
CREATE TABLE dbo.TestDateTable (i int,dt datetime2(7));
import datetime, pyodbc
def write_to_database(values):
conn = pyodbc.connect("Driver={ODBC Driver 17 for SQL Server};"
"Server=xxxx.database.windows.net;"
"Database={database};"
"UID=user;"
"PWD=password;")
with conn.cursor() as cursor:
cursor.executemany("INSERT INTO dbo.TestDateTable (i,dt) VALUES (?,?);", values)
conn.commit()
return()
date = datetime.datetime.now()
values = ([1,date],[2,date])
write_to_database(values)
I am new to Python and want to understand the syntax for connecting to MS sql database.
Have successfully created the connection but want to select a conditional data.
Please suggest how can below be done in python:
import pyodbc as odbc
cursor.execute("SELECT Col1,Col2 FROM Data where date between sysdate and (sysdate-1) ;")
I have already assigned Start and end as :
End = dt.datetime.today()
Start = End - dt.timedelta(days=1)
How can this be incorporated in the query above??
Also, it would be really helpful if you could point me to some documentation where other such related things might be mentioned.
As explained in the pyodbc documentation for .execute, you can use ? as parameter placeholders in your SQL command text and pass the parameter values to .execute like so:
End = dt.datetime.today()
Start = End - dt.timedelta(days=1)
sql = "SELECT [Col1],[Col2] FROM [Data] WHERE [date] BETWEEN ? AND ?"
cursor.execute(sql, Start, End)
rows = cursor.fetchall()
I believe Gord's answer is the correct one but in case this is useful, you can also prepare dynamically you SQL statement within your Python code and provide the SQL statement already updated with the parameters to your cursor.
This should solve your error:
Error: ('HYC00', '[HYC00] [Microsoft][ODBC SQL Server Driver]Optional
feature not implemented (0) (SQLBindParameter)')
end = dt.datetime.today()
start = End - dt.timedelta(days=1)
sql = "SELECT [Col1],[Col2] FROM [Data] WHERE [date] BETWEEN '{}' AND '{}'".format(start, end)
cursor.execute(sql)
rows = cursor.fetchall()
Start = StringVar()
End = StringVar()
End = dt.datetime.today()
Start = End - dt.timedelta(days=1)
sql = "SELECT [Col1],[Col2] FROM [Data] WHERE [date] BETWEEN ? AND ?"
cursor.execute(sql, Start, End)
rows = cursor.fetchall()
I follow this link to query Azure database.
import pyodbc
server = 'your_server.database.windows.net'
database = 'your_database'
username = 'your_username'
password = 'your_password'
driver= '{ODBC Driver 13 for SQL Server}'
cnxn = pyodbc.connect('DRIVER='+driver+';PORT=1433;SERVER='+server+';PORT=1443;DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
cursor.execute("SELECT * FROM FinancialRecord where deleted=0")
row = cursor.fetchone()
while row:
print (str(row[0]) + " " + str(row[1]))
row = cursor.fetchone()
When I run the code above, it show the error.
Traceback (most recent call last):
File "sqltest.py", line 10, in
row = cursor.fetchone()
pyodbc.ProgrammingError: ('ODBC SQL type -155 is not yet supported. column-index=2 type=-155', 'HY106')
I am new to Azure. Anyone can help?
pyodbc supports Output Converter functions that we can use when a database returns an SQL type that pyodbc does not support natively. The example on the Wiki page linked above will perform a client-side conversion similar to what would be achieved by a CAST to [N]VARCHAR on the server:
import struct
import pyodbc
conn = pyodbc.connect("DSN=myDb")
def handle_datetimeoffset(dto_value):
# ref: https://github.com/mkleehammer/pyodbc/issues/134#issuecomment-281739794
tup = struct.unpack("<6hI2h", dto_value) # e.g., (2017, 3, 16, 10, 35, 18, 0, -6, 0)
tweaked = [tup[i] // 100 if i == 6 else tup[i] for i in range(len(tup))]
return "{:04d}-{:02d}-{:02d} {:02d}:{:02d}:{:02d}.{:07d} {:+03d}:{:02d}".format(*tweaked)
crsr = conn.cursor()
# create test data
crsr.execute("CREATE TABLE #dto_test (id INT PRIMARY KEY, dto_col DATETIMEOFFSET)")
crsr.execute("INSERT INTO #dto_test (id, dto_col) VALUES (1, '2017-03-16 10:35:18 -06:00')")
conn.add_output_converter(-155, handle_datetimeoffset)
value = crsr.execute("SELECT dto_col FROM #dto_test WHERE id=1").fetchval()
print(value)
crsr.close()
conn.close()
which prints
2017-03-16 10:35:18.0000000 -06:00
ODBC SQL type -155 corresponds to the SQL Server type DatetimeOFFSET and the ODBC type SQL_SS_TIMESTAMPOFFSET. The mapping between ODBC types and SQL Server types is described in this documentation page. The error message says that this SQL Server datatype is currently unsupported by the Python ODBC API.
To work around the issue, you will need to change your query to avoid querying columns with DatetimeOFFSET datatype. One way to proceed is to identify the columns in your FinancialRecord table that have a DatetimeOFFSET datatype and convert them to a nvarchar(100) type.
SELECT CAST(MyColumn as AS nvarchar(100)) AS MyColumnAsNVarChar', ...
FROM FinancialRecord where deleted=0
I'm using pyodbc to query to an SQL Server database
import datetime
import pyodbc
conn = pyodbc.connect("Driver={SQL Server};Server='dbserver',Database='db',
TrustedConnection=Yes")
cursor = conn.cursor()
ratings = ("PG-13", "PG", "G")
st_dt = datetime(2010, 1, 1)
end_dt = datetime(2010, 12, 31)
cursor.execute("""Select title, director, producer From movies
Where rating In ? And release_dt Between ? And ?""",
ratings, str(st_dt), str(end_dt))
but am receiving the error below. Does the tuple parameter need to be handled in a different way? Is there a better way to structure this query?
('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Line 9:
Incorrect syntax near '#P1'. (170) (SQLExecDirectW);
[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]
Statement(s) could not be prepared. (8180)")
UPDATE:
I was able to get this query to work using the string formatting operator, which isn't ideal as it introduces security concerns.
import datetime
import pyodbc
conn = pyodbc.connect("Driver={SQL Server};Server='dbserver',Database='db',
TrustedConnection=Yes")
cursor = conn.cursor()
ratings = ("PG-13", "PG", "G")
st_dt = datetime(2010, 1, 1)
end_dt = datetime(2010, 12, 31)
cursor.execute("""Select title, director, producer From movies
Where rating In %s And release_dt Between '%s' And '%s'""" %
(ratings, st_dt, end_dt))
To expand on Larry's second option - dynamically creating a parameterized string, I used the following successfully:
placeholders = ",".join("?" * len(code_list))
sql = "delete from dbo.Results where RESULT_ID = ? AND CODE IN (%s)" % placeholders
params = [result_id]
params.extend(code_list)
cursor.execute(sql, params)
Gives the following SQL with the appropriate parameters:
delete from dbo.Results where RESULT_ID = ? AND CODE IN (?,?,?)
You cannot parameterize multiple values in an IN () clause using a single string parameter. The only way to accomplish that is:
String substitution (as you did).
Build a parameterized query in the form IN (?, ?, . . ., ?) and then pass in a separate parameter for each place holder. I'm not an expert at Python to ODBC but I imagine that this is particularly easy to do in a language like Python. This is safer because you get the full value of parameterization.
To expand on Larry and geographika's answers:
ratings = ('PG-13', 'PG', 'G')
st_dt = datetime(2010, 1, 1)
end_dt = datetime(2010, 12, 31)
placeholders = ', '.join('?' * len(ratings))
vars = (*ratings, st_dt, end_dt)
query = '''
select title, director, producer
from movies
where rating in (%s)
and release_dt between ? and ?
''' % placeholders
cursor.execute(query, vars)
With the placeholder, this will return a query of:
select title, director, producer
from movies
where rating in (?, ?, ?)
and release_dt between ? and ?
If you pass in ratings, it'll attempt to fit all of its items into one ?. However, if we pass in *ratings, and each item in ratings will take its place in the in() clause. Thus, we pass the tuple (*ratings, st_dt, end_dt) to cursor.execute().
The problem is your tuple. The ODBC connection is expecting a string to construct the query and you are sending a python tuple. And remember that you have to get the string quoting correct. I'm assuming that the number of ratings you will be looking for varies. There is probably a better way, but my pyodbc tends to be simple and straightforward.
Try the following:
import datetime
import pyodbc
conn = pyodbc.connect("Driver={SQL Server};Server='dbserver',Database='db',
TrustedConnection=Yes")
def List2SQLList(items):
sqllist = "%s" % "\",\"".join(items)
return sqllist
cursor = conn.cursor()
ratings = ("PG-13", "PG", "G")
st_dt = datetime(2010, 1, 1)
end_dt = datetime(2010, 12, 31)
cursor.execute("""Select title, director, producer From movies
Where rating In (?) And release_dt Between ? And ?""",
List2SQLList(ratings), str(st_dt), str(end_dt))