I have this table structure:
date_sourced
sha1
vsdt
trendx
notes
And my csv structure: sha1,vsdt,trendx,notes
How can I insert a variable value to my date_sourced?
I tried this:
var = "2018-1-10"
query = "LOAD DATA INFILE %s INTO TABLE jeremy_table_test FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (%s,sha1, #var1, trendx,notes) SET vsdt = TRIM(TRAILING ')' FROM TRIM(LEADING '(' FROM #var1))"
cursor.execute(query, (path,var))
but gives me error:
ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''2018-1-10',sha1, #var1, trendx,notes) SET vsdt = TRIM(TRAILING ')' FROM TRIM(LE' at line 1
Does LOAD_DATA_INFILE accept external variables? For example I have these two variables
import csv
import mysql.connector
path = 'C:\\Users\\trendMICRO\\Desktop\\OJT\\updated_test.csv'
print "CSV importing to database"
mydb = mysql.connector.connect(user='root', password='',
host='localhost',
database='jeremy_db')
cursor = mydb.cursor()
var = "apple"
query = "LOAD DATA INFILE %s INTO TABLE jeremy_table_test FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (%s, #var1, person) SET vsdt = TRIM(TRAILING ')' FROM TRIM(LEADING '(' FROM #var1))"
cursor.execute(query, (path))
mydb.commit()
How can I apply it here in my query,by replacing 'path/to/rb' to my variable path and value of fruit set by my variable var = "apple"?
LOAD DATA INFILE 'path/to/rb' INTO TABLE jeremy_table_test FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (fruit, #var1, person) SET vsdt = TRIM(TRAILING ')' FROM TRIM(LEADING '(' FROM #var1))
cursor.execute(query)
connection.commit()
This answer assumes you are using the MySQLdb module, if you are using a different driver the answer may vary.
To add the values we want to use a parameterized query as follows:
var = "apple"
path = "C:\\apple.txt"
query = "LOAD DATA INFILE %s INTO TABLE jeremy_table_test FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (%s, #var1, person) SET vsdt = TRIM(TRAILING ')' FROM TRIM(LEADING '(' FROM #var1))"
cursor.execute(query, (path, var))
connection.commit()
The parameters stored in the tuple given as the second argument to cursor.execute() will be substituted for the values of %s that occur in the query string.
Unfortunately this won't work because path is not going to be used as a column value, and according to the MySQLdb user's guide:
Parameter placeholders can only be used to insert column values. They
can not be used for other parts of SQL, such as table names,
statements, etc.
So we need to do a horrible thing and write our query string by hand using the file name. This is unsafe if you are allowing user input to be passed in to a variable like path.
We can still use a parameter for the value of var as before.
var = "apple"
path = "C:\\apple.txt"
query = "LOAD DATA INFILE '" + path + "' "
query += "INTO TABLE jeremy_table_test FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES (%s, #var1, person) SET vsdt = TRIM(TRAILING ')' FROM TRIM(LEADING '(' FROM #var1))"
cursor.execute(query, (var,))
connection.commit()
Related
I am trying to import txt file into sql, but i have an error:
TypeError: can only concatenate str (not "list") to str
My code:
import psycopg2
con = psycopg2.connect(
host = "",
database="",
user = "",
password = "")
cursor = con.cursor()
with open("pom1.txt") as infile:
for line in infile:
data = line.split()
print(data)
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES"
"(" + data + ");")
cursor.execute(query, *data)
con.commit()
Does anyone have an idea how can i solve it? :)
You don't put the actual values into the parameterized query; you put whatever placeholders are appropriate for your library.
data = line.split()
place_holders = ', '.join("%s" for _ in data) # Assuming %s is correct
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES"
"(" + place_holders + ");")
cursor.execute(query, *data)
cursor.execute takes care of inserting each value where a placeholder occurs, ensure things are properly quoted/escaped/etc.
There are several problems here. First, as the error says, you are trying to concatenate a List (which is data) directly to a string.
Second, you should not use + to concatenate your values and your query.
The doc says:
Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
You should only pass the values to the query via %s.
I'm not sure about the use of * in front of data in cursor.execute(query, *data).
Here is a code that should work, though I have nothing at hand for testing it right now:
import psycopg2
con = psycopg2.connect(
host = "",
database="",
user = "",
password = "")
cursor = con.cursor()
with open("pom1.txt") as infile:
for line in infile:
data = line.split()
print(data)
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES (%s, %s, %s, %s);")
cursor.execute(query, data)
con.commit()
I am using python 3.6 and psycopg2 to upload CSV-files to a postgres database. postgresql is not too happy with quotes and paranthesis in the variables. Is there a smart way to insert such variables into a database?
var_list = ['p_pladser.3320', '1108', "Christian II's Allé", '1', 'nej', "Christian II's Allé", 'Ulige husnr.', 'Amager Vest', '', 'Uafmærket parkering', '2012-02-14T12:06:07', '2009-07-15T00:00:00', '', '37086', 'MULTILINESTRING ((12.60868230570311 55.65583969695316, 12.608588075325498 55.65581925066134))']
I have tried
query = "INSERT INTO p_pladser (FID, vejkode, vejnavn, antal_pladser, restriktion, vejstatus, vejside, bydel, p_ordning, p_type, rettelsedato, oprettelsesdato, bemaerkning, id, wkb_geometry) VALUES %s" % repr(tuple(map(str,var_list)))
dbcur.execute(query)
and
query = "INSERT INTO p_pladser (FID, vejkode, vejnavn, antal_pladser, restriktion, vejstatus, vejside, bydel, p_ordning, p_type, rettelsedato, oprettelsesdato, bemaerkning, id, wkb_geometry) VALUES %s" % ','.join('?' * len(var_list))
cursor.execute(query, var_list)
Both suggestions from another post to a similar but simple problem.
In short: don't put literals into the query; use placeholders and parameter binding.
I found out that in SQL you escape ' by adding an extra '. The query works now if I just do the following.
# Replacing single quote with two single quotes
var_list = [w.replace("'", "''") for w in var_list]
# Adding the variables to query
query = "INSERT INTO p_pladser (FID, vejkode, vejnavn, antal_pladser, restriktion, vejstatus, vejside, bydel, p_ordning, p_type, rettelsedato, oprettelsesdato, bemaerkning, id, wkb_geometry) VALUES (" + (', '.join("'" + item + "'" for item in row)) + ")"
# Executing query
dbcur.execute(query)
I am trying to insert my data from .csv to Mysql database using python script.
python script which i used
import csv
import MySQLdb
db = MySQLdb.connect(host='localhost',user='root',passwd='password',db='EfficientBazzar')
cursor = db.cursor()
csv_data = csv.reader(file('products.csv'))
for row in csv_data:
cursor.execute('INSERT INTO vendor_price_list(ID,Vendor,productname,productcode,unit,weight,price)' 'VALUES("%s","%s","%s","%s","%s","%s","%s")')
db.commit()
cursor.close()
print "Done"
You're forgetting to insert the row data into your update query.
Replace:
'VALUES("%s","%s","%s","%s","%s","%s","%s")'
with:
'VALUES("%s","%s","%s","%s","%s","%s","%s")' % tuple(row)
Actually even better, consider parameterizing query:
cursor.execute('INSERT INTO vendor_price_list(ID,Vendor,productname,productcode,unit,weight,price)' + \
' VALUES(%s,%s,%s,%s,%s,%s,%s)', row)
Even faster is a bulk command straight from csv with MySQL's LOAD DATA INFILE (assuming server instance settings allow it):
cursor.execute("LOAD DATA INFILE '/path/to/products.csv'" + \
" INTO TABLE vendor_price_list" + \
" FIELDS TERMINATED BY ',' ENCLOSED BY '\"'" + \
" LINES TERMINATED BY '\n' IGNORE 1 ROWS")
I am trying to get the mssql table column names using pyodbc, and getting an error saying
ProgrammingError: No results. Previous SQL was not a query.
Here is my code:
class get_Fields:
def GET(self,r):
web.header('Access-Control-Allow-Origin', '*')
web.header('Access-Control-Allow-Credentials', 'true')
fields = []
datasetname = web.input().datasetName
tablename = web.input().tableName
cnxn = pyodbc.connect(connection_string)
cursor = cnxn.cursor()
query = "USE" + "[" +datasetname+ "]" + "SELECT COLUMN_NAME,* FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = " + "'"+ tablename + "'"
cursor.execute(query)
DF = DataFrame(cursor.fetchall())
columns = [column[0] for column in cursor.description]
return json.dumps(columns)
how to solve this?
You can avoid this by using some of pyodbc's built in methods. For example, instead of:
query = "USE" + "[" +datasetname+ "]" + "SELECT COLUMN_NAME,* FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = " + "'"+ tablename + "'"
cursor.execute(query)
DF = DataFrame(cursor.fetchall())
Try:
column_data = cursor.columns(table=tablename, catalog=datasetname, schema='dbo').fetchall()
print(column_data)
That will return the column names (and other column metadata). I believe the column name is the fourth element per row. This also relieves the very valid concerns about SQL injection. You can then figure out how to build your DataFrame from the resulting data.
Good luck!
Your line
query = "USE" + "[" +datasetname+ "]" + "SELECT COLUMN_NAME,*...
Will produce something like
USE[databasename]SELECT ...
In SSMS this would work, but I'd suggest to look on proper spacing and to separate the USE-statement with a semicolon:
query = "USE " + "[" +datasetname+ "]; " + "SELECT COLUMN_NAME,*...
Set the database context using the Database attribute when building the connection string
Use parameters any time you are passing user input (especially from HTTP requests!) to a WHERE clause.
These changes eliminate the need for dynamic SQL, which can be insecure and difficult to maintain.
I am trying to update an entry in a table usinig Python cx_oracle. The column is named "template" and it has a data type of CLOB.
This is my code:
dsn = cx_Oracle.makedsn(hostname, port, sid)
orcl = cx_Oracle.connect(username + '/' + password + '#' + dsn)
curs = orcl.cursor()
sql = "update mytable set template='" + template + "' where id='6';"
curs.execute(sql)
orcl.close()
When I do this, I get an error saying the string literal too long. The template variable contains about 26000 characters. How do I solve this?
Edit:
I found this: http://osdir.com/ml/python.db.cx-oracle/2005-04/msg00003.html
So I tried this:
curs.setinputsizes(value = cx_Oracle.CLOB)
sql = "update mytable set template='values(:value)' where id='6';"
curs.execute(sql, value = template)
and I get a "ORA-01036: illegal variable name/number error"
Edit2:
So this is my code now:
curs.setinputsizes(template = cx_Oracle.CLOB)
sql = "update mytable set template= :template where id='6';"
print sql, template
curs.execute(sql, template=template)
I get an ORA-00911: invalid character error now.
Inserting values in sql statements is a very bad practice. You should use parameters instead:
dsn = cx_Oracle.makedsn(hostname, port, sid)
orcl = cx_Oracle.connect(username + '/' + password + '#' + dsn)
curs = orcl.cursor()
curs.setinputsizes(template = cx_Oracle.CLOB)
sql = "update mytable set template= :template where id='6'"
curs.execute(sql, template=template)
orcl.close()
Use IronPython
import sys
sys.path.append(r"...\Oracle\odp.net.11g.64bit")
import clr
clr.AddReference("Oracle.DataAccess")
from Oracle.DataAccess.Client import OracleConnection, OracleCommand, OracleDataAdapter
connection = OracleConnection('userid=user;password=hello;datasource=database_1')
connection.Open()
command = OracleCommand()
command.Connection = connection
command.CommandText = "SQL goes here"
command.ExecuteNonQuery()
Change your table definition. A varchar2 field can store up to 32767 bytes; so, if you're using an 8-bit encoding, you have a bit of room left to play with before resorting to LOBs.