I'm trying to debug a problem inserting data into an a Microsoft SQL database. The error message I keep getting is:
ProgrammingError: ('42000', "[42000] [Microsoft][ODBC SQL Server
Driver][SQL Server]The incoming Tabular Data Stream (TDS) for the
Remote Procedure Call (RPC) protocol is not correct. Parameter 16
(''): The specified value is not a valid instance of the float data
type. Check the source data for invalid values. An example of an
invalid value is data of a numeric type whose decimal places exceed
the precision value. (8023) (SQLExecDirectW)")
I (believe) I understand the problem but I can't isolate which values are tripping it. I made sure all the columns in the dataframe are exactly the same name and order of the database, but obviously something still isn't jiving. My question is: Where exactly is this error message pointing too when it says 'Parameter 16'. Is is a row? Is it a column? Is it zero-indexing?
I unfortunately can't easily provide data or examples but I can at least provide my sql insert code for more context:
with engine.begin() as conn:
stmt = insert(reflection)
values_list = df.to_dict(orient='records')
conn.execute(stmt, values_list)
Any tips?
Related
I am using pyodbc to connect to a local MS SQL Server 2019 instance on Windows. There is a table in a database on that instance called 'Media', which has a column called 'mediaId', which has a bigint datatype. For reference, it is a column that is uniquely constrained, but is neither a primary nor a foreign key. I am trying to update specific records in this column with pyodbc.
For reference, the following code executes successfully in Python using pyodbc.
db = DB() # db contains pyodbc context
cursor = db.cursor
command = "UPDATE Media SET {}={} WHERE id=476506567332605952".format('mediaId', 476506524101914624)
cursor.execute(command)
What does not execute cleanly is the following snippet.
db = DB()
cursor = db.cursor
command = "UPDATE Media SET ?=? WHERE id=476506567332605952"
params = ('mediaId', 476506524101914624)
cursor.execute(command, params)
As far as I am aware, the parameters should be inserted where the question marks are in the same way as the statement above. However, this yields the following error message.
[22003] [Microsoft][ODBC Driver 17 for SQL Server][SQL
Server]Arithmetic overflow error converting expression to data type
nvarchar. (8115) (SQLExecDirectW); [22003] [Microsoft][ODBC Driver 17
for SQL Server][SQL Server]The statement has been terminated. (3621)
No nvarchar types are used in this table, and the integer that is to be set falls under the bigint size limit and under the int limit for a 64-bit system in Python.
I am at a loss as to why I might be getting this error.
The issue is that you are using ? substitution for the column name. The database adapter will quote the value as if it were a data column, so the expression becomes UPDATE Media SET (string)=(integer), and I'm guessing the parser tries to coerce the integer to a string to compare it.
Don't use ? substitution for column and table names.
I get error using python 3.8.5
( cursor.execute("INSERT INTO dbo.sftpserverlist(FileName,FileSize) VALUES ("+files[0]+","+str(sizes[0])+")")
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]The multi-part identifier "DidosSupply.zip" could not be bound. (4104) (SQLExecDirectW)')
while I am trying call following function and insert into table dbo.sftpserverlist.
def getfile(sftp):
FileNames=[]
FileName = sftp.listdir_attr()
for i in FileName:
FileNames.append(i.filename)
FileSizes=[]
FileSize = sftp.listdir_attr()
for i in FileSize:
FileSizes.append(i.st_size)
return FileNames,FileSizes
-----------------------------------------------------------
cursor.execute("INSERT INTO dbo.sftpserverlist(FileName,FileSize) VALUES ("+files[0]+","+str(sizes[0])+")")
conn.commit()
Parametrise your query, don't inject the values, which is a huge security flaw.
cursor.execute("INSERT INTO dbo.sftpserverlist(FileName,FileSize) VALUES (?,?)",files[0],str(sizes[0]))
conn.commit()
cursor.close()
The reason you were getting the error was because the values you were (insecurely) injecting weren't quoted; thus the value of files[0] (which in this case had the value "DidosSupply.zip") was being interpreted as a column's name; which it can't be as you're within a VALUES clause with no FROM.
Of course, just wrapping quotes around the value isn't the solution, as quotes can be escaped.
I'm not an expert with Python, by any means, but I think this is the old-school way of doing it (INSERT INTO). I recently stumbled upon a super-easy, scalable, and controllable, way of pushing data from Python to SQL Server. Try the sample code and post back if you have additional questions.
import pyodbc
import pandas as pd
engine = "mssql+pyodbc://your_server_name/your_database_name?driver=SQL Server Native Client 11.0?trusted_connection=yes"
... dataframe here...
dataframe.to_sql(x, engine, if_exists='append', index=True)
dataframe is pretty self explanatory.
x = the name yo uwant your table to be in SQL Server.
i am a novice at python and sql so i have encounterd some issue while trying to create a stored procedure and passing values to it.
i have looked through few questions here.Of what i could understand i used the ODBC format and it works for procedures not requiring anything to be passed. The query is simple. it takes in 2 parameters route_no and month. the query then count the number of trios taken by a bus per date for that month on that route.
if __name__ == "__main__":
conn=pyodbc.connect('''Driver={SQL Server};'''
'''Server=ABY;'''
'''Database=testing;'''
'''Trusted_Connection=yes;''')
cursor=conn.cursor()
proc3="""CREATE PROCEDURE TRIPS_PER_DAY #ROUTENO NVARCHAR(30),#MONTH NVARCHAR(30)
AS
BEGIN
SELECT [ON DATE],SUM(DISTINCT([TRIP NO])) AS SUM_TRIPS
FROM testing.dbo.sheet1$
WHERE [ROUTE NO#]=#ROUTENO AND MONTH([ON DATE])=#MONTH
GROUP BY [ON DATE]
END"""
drop_proc3="""IF EXIST(SELECT * FROM SYS.OBJECTS WHERE TYPE='P' AND NAME='TRIPS_PER_DAY') DROP PROCEDURE TRIPS_PER_DAY"""
call_proc3="""{CALL testing.dbo.TRIPS_PER_DAY(?)(?)}"""
cursor.execute(drop_proc3)
cursor.execute(proc3)
values=('16','11-11-2017')
dff=pd.read_sql(call_proc3,conn,params=(values,))
print(dff)
this is the error i am getting:
DatabaseError: Execution failed on sql '{CALL testing.dbo.PASS_PER_STOP(?)(?)}': ('42000', '[42000] [Microsoft][ODBC SQL Server Driver]Syntax error or access violation (0) (SQLPrepare)')
i am not sure if the method i am using is right or if there is a syntax error.
I am also not sure how to pass values and return results from the procedure using python. Any help?. Thanks in advance!!!
I keep running into an error and cannot seem to find a solution anywhere online. I currently have a SQL that has 36 columns and am using a delete statement then inserting new values each day via a CSV and python program. It is a basic Insert Into statement that is utilizing a python for loop to insert all rows from the CSV.
I just added a new column to the SQL table and to my insert statement within the python program (I have done this multiple times to this same table in the past), however whenever I the new insert statement program I get the following error:
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC SQL Server
Driver][SQL Server]Incorrect syntax near '#P26'. (102) (SQLExecDirectW);
[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could
not be prepared. (8180)")
I assume the '#P26' is referring to some parameter, however I am not utilizing any SQL Parameters. Below is a snippet of my Python / SQL code:
SQL = """insert into example_table( [Column_1],
[Column_2],
[Column_3],
[Column_4],
[Column_5],
[Column_6],
[Column_7],
[Column_8],
[Column_9],
[Column_10],
....
[Column_36],
[New Column]
) values (?,?,?,?,?,?,?,?,?,?,?,?,
?,?,?,?,?,?,?,?,?,?,?,?,
?,?,?,?,?,?,?,?,?,?,?,?,?)"""
cursor2.execute("delete example_table")
for row in csv_data:
cursor2.execute(SQL, row)
Thanks so much for the help I am completely stuck. Sorry for the weird indents in the code. (NOTE: I know the syntax is correct as it works when I delete the new column and parameter marker and re-run).
This is a syntax issue, please check your "?" 26th value. This happen to me i has this syntax:
.... ?,?,?.?)
this was my 23th value and the error told me #P23.
Source: exp.
I have a python script that makes a call to an API, submits a request, and then is supposed to insert the result into a Sql Server 2012 table. When it goes to execute the insert into SQL, it breaks. I am currently importing json, requests, and pyodbc into the file. Here is the location where it is breaking:
conn = pyodbc.connect('DRIVER={SQL Server};SERVER={localServer};DATABASE={localDB}')
cursor = conn.cursor()
for record in response:
print(json.dumps(record))
cursor.execute("Insert Into Ticket_Info values ?", json.dumps(record))
cursor.commit()
cursor.close()
conn.close()
It is at the cursor.execute() line where the breakage occurs. This is the error I got when I attempted to run this.
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC SQL
Server Driver][SQL Server]Incorrect syntax near '#P1'. (102)
(SQLExecDirectW); [42000] [Microsoft][ODBC SQL Server Driver][SQL
Server]Statement(s) could not be prepared. (8180)"
Any help I could get I would appreciate. I have searched and tried several different methods at this point, the only thing that changes is the error.
The second argument to cursor.execute() must be a sequence of values to interpolate, one for each SQL parameter in your statement.
You gave ODBC a string instead, which is also a sequence, but one that contains (many) more elements (characters) than your query requires.
Use a single-element tuple here:
cursor.execute("Insert Into Ticket_Info values (?)", (json.dumps(record),))
I also put parenthesis around the values section, as per the SQL Server INSERT syntax:
VALUES
Introduces the list or lists of data values to be inserted. There must be one data value for each column in column_list, if specified, or in the table. The value list must be enclosed in parentheses.
Unless Ticket_Info has only one column per row (unlikely, you'd have a primary key column at least), you probably need to specify what column you are inserting your value into:
cursor.execute("Insert Into Ticket_Info (<columnname>) values (?)", (json.dumps(record),))
where you need to replace <columnname> with the actual column name in your table.