We Keep Coding

while inserting records in mysql, argument values are passing as 0 in python

In console.log I am getting the correct argument value, but when I try to add the argument value in the insert statement it is passing as 0. As a whole, the code works fine without errors, but for assg_id, instead of the actual value, it is inserting 0.
import pandas as pd
import sys
print ("parameters from nodejs", str(sys.argv[1]))
df = pd.read_csv("./userSetupData.csv")
df.head()
import mysql.connector as msql
from mysql.connector import Error
try:
conn = msql.connect(host='localhost', database='pythonTest', user='root', password='0000', auth_plugin='mysql_native_password')
if conn.is_connected():
cursor = conn.cursor()
cursor.execute("select database();")
record = cursor.fetchone()
assgn_id = str(sys.argv[1])
print('Checking the parameter value', assgn_id)
for i,row in df.iterrows():
#here %S means string values
sql = "INSERT INTO pythonTest.usr_stg VALUES (%s,%s,%s,%s,%s,assgn_id)"
cursor.execute(sql, tuple(row))
print("Record inserted")
# the connection is not auto committed by default, so we must commit to save our changes
conn.commit()
except Error as e:
print("Error while connecting to MySQL", e)

sql has normal string and it treats assgn_id as normal text in string, not Python's variable
You would have to use f-string and "{assgn_id}" to put value in this string
sql = f"INSERT INTO pythonTest.usr_stg VALUES (%s,%s,%s,%s,%s, {assgn_id})"
Or you should put assgn_id in tuple with parameters (and use %s in query)
sql = "INSERT INTO pythonTest.usr_stg VALUES (%s,%s,%s,%s,%s, %s)"
cursor.execute(sql, tuple(row.to_list() + [assgn_id]) )

Python: malformed array literal when inserting a string

I'm trying to read from a file and insert the data to a postgresql table in python using the psycopg2.
Here's the function I wrote:
def insert(file, table, col, conn):
sql = "INSERT INTO "+table+"("+col+") VALUES(%s)"
cur = conn.cursor()
with open(os.path.join(DEFAULTS_FOLDER, file)) as fp:
line = fp.readline()
while line:
cur.execute(sql, (line.rstrip(),))
line = fp.readline()
conn.commit()
cur.close()
return
For some reason I get an error:
cur.execute(sql, (line.rstrip(),)) psycopg2.DataError: malformed array literal: "hello" LINE 1: INSERT INTO greetings(gname) VALUES('hello')
I also tried to insert a plain string and I still get the same error.

The error message means that the column gname of the table greetings is an array, not a plain text. If it is a text array, the query should look like this:
INSERT INTO greetings(gname) VALUES('{hello}')
You should change the relevant fragment of your code, e.g.:
cur.execute(sql, ("{{{}}}".format(line.rstrip()),))

Decode a string: Python

I built a string with a tuple like this:
t = tuple(data)
querysring="INSERT INTO %s VALUES %s "%(table,t)
When I print the string the result is:
INSERT INTO AGENT VALUES ('Bock', 'Fran\\xc3\\xa7ois Bock', 'Individual', 'fb****#mail.com')
But I want something like this:
INSERT INTO AGENT VALUES ('Bock', 'François Bock', 'Individual', 'fb****#mail.com')
It is possible to decode the string ?
I'm using Python2.x but I can use Python3.x
I try this:
querysring=u"INSERT INTO %s VALUES %s "%(table,t)
print(ftfy.fix_text(querysring))
But it's not working

I think your issue is superficial and related to how print displays lists and list items differently. The printed output of a list is in ascii even if the items within the list are correctly encoded in utf-8. First, using chardet library:
from chardet.universaldetector import UniversalDetector
a = ['Bock', 'François Bock']
detector = UniversalDetector()
detector.feed(str(a))
detector.close()
print "Encoding for the str(list): ", detector.result
detector = UniversalDetector()
detector.feed(a[1])
detector.close()
print "Encoding for list[1]: ", detector.result
print "The whole list: ", a
print "Item in list: ", a[1]
Aside from the off-putting printouts, it's possible to still write to the database with the correct encoding with a parameterized query. The last part of the below code writes to a file to confirm that the data encoding is preserved:
import sqlite3
conn = sqlite3.connect(":memory:")
conn.text_factory = str
c = conn.cursor()
c.execute("CREATE TABLE IF NOT EXISTS testing(test1 TEXT, test2 TEXT)")
conn.commit()
my_tuple = 'Bock', 'François Bock'
table = 'testing'
placeholders = ', '.join('?' for item in my_tuple)
query = "INSERT INTO {} VALUES ({})".format(table, placeholders)
c.execute(query, my_tuple)
c.execute("SELECT * FROM testing")
all_data = c.fetchone()
# Check the printouts
print all_data
print all_data[1]
# For good measure, write them to a file
with open('check_output.txt', 'w') as outfile:
outfile.write(', '.join(item for item in all_data))

Read from csv file and put to Mysql with different field type using python

i have table field of "datetime", "integer", "double", "string (varchar)" type in Mysql, how can i convert from string (from csv file) to each type of Mysql field using python ? example of my csv file :
EURUSD;M5;2011.01.05 02:10:00;1.33193;1.33193;1.33112;1.33135;0;-1;0;0;1.33215
My python code is like this :
import MySQLdb
# Open database connection
db = MySQLdb.connect("localhost","me","xxxxx","test" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
inputFile = "source.csv"
with open(inputFile, 'r') as fr:
data = fr.readlines()
for line in data:
words = line.strip().split(";")
getpair = words[0]
gettf = words[1]
gettime = (datetime) words[2]
getopen = (double) words[3]
gethigh = (double) words[4]
getlow = (double) words[5]
getclose = (double) words[6]
getcUpdown = (int) words[7]
getfractal = (int) words[8]
getzzId = (int) words[9]
getzzUpdown = (int) words[10]
getzzEndPrc = (double) words[11]
print(getpair,"=",gettf,"=",gettime,"=",getopen,"=",gethigh,"=",getlow,"=",getclose,"=",getcUpdown,"=",getfractal,"=",getzzId,"=",getzzUpdown,"=",getzzEndPrc)
# =====================================================================
# Prepare SQL query to INSERT a record into the database.
sql = '''INSERT INTO `mytable` (pair,timeframe,time,open,close,high,low,updown,fractal,zzid,zzupdown,zzlength) \
VALUES (getpair,gettf,gettime,getopen,getclose,gethigh,getlow,getcUpdown,getfractal,getzzId,getzzUpdown,getzzEndPrc)'''
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
fr.close()
Thanks
p

Type conversion in Python is done by:
int(words[7])
not
(int)words[7]
There is no double type in Python. Use float instead.
You can not convert directly to datetime. Use datetime.strptime() to parse the input string:
from datetime import datetime
gettime = datetime.strptime(words[2], '%Y.%m.%d %H:%M:%S')
You might not need to do this (conversion should be handled when using parameterised queries - see below), but you can convert that to a string format that MySQLDB supports. You can use strftime() for that:
gettime = gettime.strftime('%Y-%m-%d %H:%M:%S')
The sql query string needs to substitute the the actual values. Your
query is inserting the names of the variables, not the
values. You can (and should) use a parameterised query like this:
sql = 'INSERT INTO `mytable` (pair, timeframe, time, open, close, high, low, updown, fractal, zzid, zzupdown, zzlength) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)'
cursor.execute(sql, (getpair, gettf, gettime, getopen, getclose, gethigh, getlow, getcUpdown, getfractal, getzzId, getzzUpdown, getzzEndPrc))
cursor.execute(sql, ...) needs to be moved into the body of the for loop.

Inserting JSON into MySQL using Python

I have a JSON object in Python. I am Using Python DB-API and SimpleJson. I am trying to insert the json into a MySQL table.
At moment am getting errors and I believe it is due to the single quotes '' in the JSON Objects.
How can I insert my JSON Object into MySQL using Python?
Here is the error message I get:
error: uncaptured python exception, closing channel
<twitstream.twitasync.TwitterStreamPOST connected at
0x7ff68f91d7e8> (<class '_mysql_exceptions.ProgrammingError'>:
(1064, "You have an error in your SQL syntax; check the
manual that corresponds to your MySQL server version for
the right syntax to use near ''favorited': '0',
'in_reply_to_user_id': '52063869', 'contributors':
'NULL', 'tr' at line 1")
[/usr/lib/python2.5/asyncore.py|read|68]
[/usr/lib/python2.5/asyncore.py|handle_read_event|390]
[/usr/lib/python2.5/asynchat.py|handle_read|137]
[/usr/lib/python2.5/site-packages/twitstream-0.1-py2.5.egg/
twitstream/twitasync.py|found_terminator|55] [twitter.py|callback|26]
[build/bdist.linux-x86_64/egg/MySQLdb/cursors.py|execute|166]
[build/bdist.linux-x86_64/egg/MySQLdb/connections.py|defaulterrorhandler|35])
Another error for reference
error: uncaptured python exception, closing channel
<twitstream.twitasync.TwitterStreamPOST connected at
0x7feb9d52b7e8> (<class '_mysql_exceptions.ProgrammingError'>:
(1064, "You have an error in your SQL syntax; check the manual
that corresponds to your MySQL server version for the right
syntax to use near 'RT #tweetmeme The Best BlackBerry Pearl
Cell Phone Covers http://bit.ly/9WtwUO''' at line 1")
[/usr/lib/python2.5/asyncore.py|read|68]
[/usr/lib/python2.5/asyncore.py|handle_read_event|390]
[/usr/lib/python2.5/asynchat.py|handle_read|137]
[/usr/lib/python2.5/site-packages/twitstream-0.1-
py2.5.egg/twitstream/twitasync.py|found_terminator|55]
[twitter.py|callback|28] [build/bdist.linux-
x86_64/egg/MySQLdb/cursors.py|execute|166] [build/bdist.linux-
x86_64/egg/MySQLdb/connections.py|defaulterrorhandler|35])
Here is a link to the code that I am using http://pastebin.com/q5QSfYLa
#!/usr/bin/env python
try:
import json as simplejson
except ImportError:
import simplejson
import twitstream
import MySQLdb
USER = ''
PASS = ''
USAGE = """%prog"""
conn = MySQLdb.connect(host = "",
user = "",
passwd = "",
db = "")
# Define a function/callable to be called on every status:
def callback(status):
twitdb = conn.cursor ()
twitdb.execute ("INSERT INTO tweets_unprocessed (text, created_at, twitter_id, user_id, user_screen_name, json) VALUES (%s,%s,%s,%s,%s,%s)",(status.get('text'), status.get('created_at'), status.get('id'), status.get('user', {}).get('id'), status.get('user', {}).get('screen_name'), status))
# print status
#print "%s:\t%s\n" % (status.get('user', {}).get('screen_name'), status.get('text'))
if __name__ == '__main__':
# Call a specific API method from the twitstream module:
# stream = twitstream.spritzer(USER, PASS, callback)
twitstream.parser.usage = USAGE
(options, args) = twitstream.parser.parse_args()
if len(args) < 1:
args = ['Blackberry']
stream = twitstream.track(USER, PASS, callback, args, options.debug, engine=options.engine)
# Loop forever on the streaming call:
stream.run()

use json.dumps(json_value) to convert your json object(python object) in a json string that you can insert in a text field in mysql
http://docs.python.org/library/json.html

To expand on the other answers:
Basically you need make sure of two things:
That you have room for the full amount of data that you want to insert in the field that you are trying to place it. Different database field types can fit different amounts of data.
See: MySQL String Datatypes. You probably want the "TEXT" or "BLOB" types.
That you are safely passing the data to database. Some ways of passing data can cause the database to "look" at the data and it will get confused if the data looks like SQL. It's also a security risk. See: SQL Injection
The solution for #1 is to check that the database is designed with correct field type.
The solution for #2 is use parameterized (bound) queries. For instance, instead of:
# Simple, but naive, method.
# Notice that you are passing in 1 large argument to db.execute()
db.execute("INSERT INTO json_col VALUES (" + json_value + ")")
Better, use:
# Correct method. Uses parameter/bind variables.
# Notice that you are passing in 2 arguments to db.execute()
db.execute("INSERT INTO json_col VALUES %s", json_value)
Hope this helps. If so, let me know. :-)
If you are still having a problem, then we will need to examine your syntax more closely.

The most straightforward way to insert a python map into a MySQL JSON field...
python_map = { "foo": "bar", [ "baz", "biz" ] }
sql = "INSERT INTO your_table (json_column_name) VALUES (%s)"
cursor.execute( sql, (json.dumps(python_map),) )

You should be able to insert intyo a text or blob column easily
db.execute("INSERT INTO json_col VALUES %s", json_value)

You need to get a look at the actual SQL string, try something like this:
sqlstr = "INSERT INTO tweets_unprocessed (text, created_at, twitter_id, user_id, user_screen_name, json) VALUES (%s,%s,%s,%s,%s,%s)", (status.get('text'), status.get('created_at'), status.get('id'), status.get('user', {}).get('id'), status.get('user', {}).get('screen_name'), status)
print "about to execute(%s)" % sqlstr
twitdb.execute(sqlstr)
I imagine you are going to find some stray quotes, brackets or parenthesis in there.

#route('/shoes', method='POST')
def createorder():
cursor = db.cursor()
data = request.json
p_id = request.json['product_id']
p_desc = request.json['product_desc']
color = request.json['color']
price = request.json['price']
p_name = request.json['product_name']
q = request.json['quantity']
createDate = datetime.now().isoformat()
print (createDate)
response.content_type = 'application/json'
print(data)
if not data:
abort(400, 'No data received')
sql = "insert into productshoes (product_id, product_desc, color, price, product_name, quantity, createDate) values ('%s', '%s','%s','%d','%s','%d', '%s')" %(p_id, p_desc, color, price, p_name, q, createDate)
print (sql)
try:
# Execute dml and commit changes
cursor.execute(sql,data)
db.commit()
cursor.close()
except:
# Rollback changes
db.rollback()
return dumps(("OK"),default=json_util.default)

One example, how add a JSON file into MySQL using Python. This means that it is necessary to convert the JSON file to sql insert, if there are several JSON objects then it is better to have only one call INSERT than multiple calls, ie for each object to call the function INSERT INTO.
# import Python's JSON lib
import json
# use JSON loads to create a list of records
test_json = json.loads('''
[
{
"COL_ID": "id1",
"COL_INT_VAULE": 7,
"COL_BOOL_VALUE": true,
"COL_FLOAT_VALUE": 3.14159,
"COL_STRING_VAULE": "stackoverflow answer"
},
{
"COL_ID": "id2",
"COL_INT_VAULE": 10,
"COL_BOOL_VALUE": false,
"COL_FLOAT_VALUE": 2.71828,
"COL_STRING_VAULE": "http://stackoverflow.com/"
},
{
"COL_ID": "id3",
"COL_INT_VAULE": 2020,
"COL_BOOL_VALUE": true,
"COL_FLOAT_VALUE": 1.41421,
"COL_STRING_VAULE": "GIRL: Do you drink? PROGRAMMER: No. GIRL: Have Girlfriend? PROGRAMMER: No. GIRL: Then how do you enjoy life? PROGRAMMER: I am Programmer"
}
]
''')
# create a nested list of the records' values
values = [list(x.values()) for x in test_json]
# print(values)
# get the column names
columns = [list(x.keys()) for x in test_json][0]
# value string for the SQL string
values_str = ""
# enumerate over the records' values
for i, record in enumerate(values):
# declare empty list for values
val_list = []
# append each value to a new list of values
for v, val in enumerate(record):
if type(val) == str:
val = "'{}'".format(val.replace("'", "''"))
val_list += [ str(val) ]
# put parenthesis around each record string
values_str += "(" + ', '.join( val_list ) + "),\n"
# remove the last comma and end SQL with a semicolon
values_str = values_str[:-2] + ";"
# concatenate the SQL string
table_name = "json_data"
sql_string = "INSERT INTO %s (%s)\nVALUES\n%s" % (
table_name,
', '.join(columns),
values_str
)
print("\nSQL string:\n\n")
print(sql_string)
output:
SQL string:
INSERT INTO json_data (COL_ID, COL_INT_VAULE, COL_BOOL_VALUE, COL_FLOAT_VALUE, COL_STRING_VAULE)
VALUES
('id1', 7, True, 3.14159, 'stackoverflow answer'),
('id2', 10, False, 2.71828, 'http://stackoverflow.com/'),
('id3', 2020, True, 1.41421, 'GIRL: Do you drink? PROGRAMMER: No. GIRL: Have Girlfriend? PROGRAMMER: No. GIRL: Then how do you enjoy life? PROGRAMMER: I am Programmer.');

The error may be due to an overflow of the size of the field in which you try to insert your json. Without any code, it is hard to help you.
Have you considerate a no-sql database system such as couchdb, which is a document oriented database relying on json format?

Here's a quick tip, if you want to write some inline code, say for a small json value, without import json.
You can escape quotes in SQL by a double quoting, i.e. use '' or "", to enter ' or ".
Sample Python code (not tested):
q = 'INSERT INTO `table`(`db_col`) VALUES ("{k:""some data"";}")'
db_connector.execute(q)