I have the same question as asked here:
Default value for empty integer fields when importing CSV data in MySQL
I keep getting the warning "Incorrect Integer value" when importing a csv file into Mysql. I've read all the relevant questions/answers here, and the link above is my direct question. But I'm trying to implement the answer given there in Python 2.7, and I can't seem to get it working.
My code below is my attempt to implement the answer from the above post. I think the issue is the syntax for using the "Set" clause in my Load DATA Local Infile statement...I would really appreciate any help since MySQL automatically converts empty INT's to "0" and that would mess up my analysis since I want blanks to be null. My code is:
import MySQLdb
db = MySQLdb.connect(blah, blah, blah)
cursor = db.cursor()
sql = """CREATE TABLE PLAYER(
PLAYER_ID INT,
DATE DATETIME,
BATTING_AVERAGE INT
)"""
cursor.execute(sql)
statement = """LOAD DATA LOCAL INFILE 'path/file.csv'
INTO TABLE PLAYER
COLUMNS TERMINATED BY ','
SET BATTING_AVERAGE = IF(#BATTING_AVERAGE='',NULL,#BATTING_AVERAGE)
IGNORE 1 LINES;"""
The error that this code gives is:
ProgramingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right synatx to use near 'IGNORE 1 Lines at line 5")
The SET and IGNORE clauses are backwards. Try swapping them:
statement = """LOAD DATA LOCAL INFILE 'path/file.csv'
INTO TABLE PLAYER
COLUMNS TERMINATED BY ','
IGNORE 1 LINES
SET BATTING_AVERAGE = IF(#BATTING_AVERAGE='',NULL,#BATTING_AVERAGE);"""
Related
I am trying to use pyodbc to update an existing MS Access database table with a very long multiline string. The string is actually a csv that has been turned into a string.
The query I am trying to use to update the table is as follows:
query = """
UPDATE Stuff
SET Results = '{}'
WHERE AnalyteName =
'{}'
""".format(df, analytename)
The full printed statement looks as follows:
UPDATE Stuff
SET Results =
'col a,col b,col c,...,col z,
Row 1,a1,b1,c1,
...,...,...,...,
Row 3000,a3000,b3000,c3000'
WHERE AnalyteName = 'Serotonin'
However this does not seem to be working, and I keep getting the following error:
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][ODBC Microsoft Access Driver] Syntax error in UPDATE statement. (-3503) (SQLExecDirectW)')
Which I assume is due to the format of the csv string I am trying to use to update the table with.
I have tried using INSERT and inserting a new row with the csv string and other relevant information and that seems to work. However, I need to use UPDATE as I will eventually be adding other csv strings to these columns. This leads me to believe that there is A) Something is wrong with the syntax of my UPDATE query (I am new to SQL syntax) or B) I am missing something from the documentation regarding UPDATE queries.
Is executing an UPDATE query like this possible? If so, where am I going wrong?
It would be determined by the table's field type.
For large amounts of text you'd need a blob field in your database table.
A blob field will store binary info so using blob will not 'see' illegal characters.
Answering my own question in case anyone else wants to use this.
It turns out what I was missing was brackets around the table column fields from my UPDATE statement. My final code looked something like this.
csv = df.to_csv(index=False)
name = 'some_name'
query = """
UPDATE Stuff
SET
[Results] = ?
WHERE
[AnalyteName] = ?
"""
self.cursor.execute(query, (csv, name))
I've seen several other posts here where brackets were not around the column names. However, since this is MS Access, I believe they were required for this query, or rather this specific query since it included a very long strong in the SET statement.
I welcome anyone else here to provide a more efficient method of performing this task or someone else who can provide more insight into why this is what worked for me.
I'm totally new to Python and SQL but I'm following a tutorial and I really can't get this working. Please ignore the poor string construct but I'll tidy it up when it's working. I'm trying to send two values to insert into a new row on a table, they get passed to it in the cursor execute command.
def update_height(conn, location, timestamp_height):
sql_update_height_table = """ INSERT INTO """ + str(location) + """_cushion(timestamp,height)
VALUES(?,?) """
print (sql_update_height_table)
cur = conn.cursor()
cur.execute(sql_update_height_table,timestamp_height)
conn.commit()
return cur.lastrowid
Result of print(sql_update_height_table):
INSERT INTO 0xEE738a9e_height(timestamp,height)
VALUES(?,?)
and this is the error:
sqlite3.OperationalError: near "0xEE738a9e": syntax error
I'm following this tutorial and I really can't see what I'm doing wrong, I've been looking for hours.
timestamp_height is a tuple with two entries.
I don't know why you need to use a table name like this, but 0x means as usually a HEX representation. Quote the name with backticks.
sql_update_height_table = """ INSERT INTO `""" + str(location) + """_cushion`(timestamp,height)
VALUES(?,?) """
I'm using python in TestComplete to conduct a db query, but the results seem to be empty strings and do not match the data in the table I queried. The file is a s3db file. Does that matter?
Using:
TestComplete Version 14
imported sqlite3 into python file
I've:
-Tried running the same query in SQLite. It returned the expected result
-Verified the connection is established with the correct db
---python
import sqlite3
def getInfo():
conn = sqlite3.connect(db)
c = conn.cursor()
try:
c.execute('SELECT Column_Name FROM Table_Name')
results = c.fetchall()
except:
Log.Error("Query execution failed")
for x in results:
Log.Message(x) `enter code here`
#Log.Message() works like a print statement in testcomplete.
---
Actual Output:
The program runs without errors, but the results come back as 15 lines of blank rows. 15 is the number of records within the table, so I know it's looking in the right place, but it seems like it's not identifying that there's information stored here.
Expected Output:
15 lines of data contained within the Column I specified in the query.
There is no error with sqlite3 and your DB operations. The issue is with Log.Message and what it expects as an argument. Within TestComplete, Log.Message requires variable arguments of type Variant, which can be any of the supported data types within TestComplete; String, Double/Real, Boolean, Date/Time, Object (i.e. TestComplete-recognised UI objects) and Integer.
Log.Message cannot accept arguments of the type returned by cursor.fetchall's rows.
So you'd need to convert each row into a String, e.g.
for x in results:
msg = ('{0} : {1}'.format(x[0], x[1]))
Log.Message(msg)
I'm working on a bit of python code to run a query against a redshift (postgres) SQL database, and I'm running into an issue where I can't strip off the surrounding single quotes from a variable I'm passing to the query. I'm trying to drop a number of tables from a list. This is the basics of my code:
def func(table_list):
drop_query = 'drop table if exists %s' #loaded from file
table_name = table_list[0] #table_name = 'my_db.my_table'
con=psycopg2.connect(dbname=DB, host=HOST, port=PORT, user=USER, password=PASS)
cur=con.cursor()
cur.execute(drop_query, (table_name, )) #this line is giving me trouble
#cleanup statements for the connection
table_list = ['my_db.my_table']
when func() gets called, I am given the following error:
syntax error at or near "'my_db.my_table'"
LINE 1: drop table if exists 'my_db.my_table...
^
Is there a way I can remove the surrounding single quotes from my list item?
for the time being, I've done it (what think is) the wrong way and used string concatenation, but know this is basically begging for SQL-injection.
This is not how psycopg2 works. You are using a string operator %s to replace with a string. The reason for this is to tokenize your string safely to avoid SQL injection, psycopg2 handles the rest.
You need to modify the query before it gets to the execute statement.
drop_query = 'drop table if exists {}'.format(table_name)
I warn you however, do not allow these table names to be create by outside sources, or you risk SQL injection.
However a new version of PSYCOPG2 kind of allows something similar
http://initd.org/psycopg/docs/sql.html#module-psycopg2.sql
from psycopg2 import sql
cur.execute(
sql.SQL("insert into {} values (%s, %s)").format(sql.Identifier('my_table')),[10, 20]
)
The following query causes python to crash ('python.exe has encountered a problem ...'
Process terminated with an exit code of -1073741819
The query is:
create temp table if not exists MM_lookup2 as
select lower(Album) || lower(SongTitle) as concat, ID
from MM.songs
where artist like '%;%' collate nocase
If I change from "like" to = it runs as expected, eg
create temp table if not exists MM_lookup2 as
select lower(Album) || lower(SongTitle) as concat, ID
from MM.songs
where artist = '%;%' collate nocase
I am running python v2.7.2, with whatever version of sqlite that ships in there.
The problem query runs without problem outside python.
You didn't write the database system/driver you are using. I suspect that your SQL is the problem. The % characters needs to be escaped. Possibly the db driver module tries to interpret %, and %) as format chars, and it cannot convert non-existing parameter value into a format that is acceptable by the database backend.
Can you please give us concrete Python code? Can you please try to run the same query but putting the value of '%,%' into a parameter, and pass it to the cursor as a parameter?