I am using python to insert a string into MySQL with special characters.
The string to insert looks like so:
macaddress_eth0;00:1E:68:C6:09:A0;macaddress_eth1;00:1E:68:C6:09:A1
Here is the SQL:
UPGRADE inventory_server
set server_mac = macaddress\_eth0\;00\:1E\:68\:C6\:09\:A0\;macaddress\_eth1\;00\:1E\:68\:C6\:09\:A1'
where server_name = 'myhost.fqdn.com
When I execute the update, I get this error:
ERROR 1064 (42000):
You have an error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near 'UPGRADE inventory_server
set server_mac = 'macaddress\_eth0\;00\:1E\:68\:C6\:09\' at line 1
The python code:
sql = 'UPGRADE inventory_server set server_mac = \'%s\' where server_name = \'%s\'' % (str(mydb.escape_string(macs)),host)
print sql
try:
con = mydb.connect(DBHOST,DBUSER,DBPASS,DB);
with con:
cur = con.cursor(mydb.cursors.DictCursor)
cur.execute(sql)
con.commit()
except:
return False
How can I insert this text raw?
This is one of the reasons you're supposed to use parameter binding instead of formatting the parameters in Python.
Just do this:
sql = 'UPGRADE inventory_server set server_mac = %s where server_name = %s'
Then:
cur.execute(sql, macs, host)
That way, you can just deal with the string as a string, and let the MySQL library figure out how to quote and escape it for you.
On top of that, you generally get better performance (because MySQL can compile and cache one query and reuse it for different parameter values) and avoid SQL injection attacks (one of the most common ways to get yourself hacked).
Welcome to the world of string encoding formats!
tl;dr - The preferred method for handling quotes and escape characters when storing data in MySQL columns is to use parameterized queries and let the MySQLDatabase driver handle it. Alternatively, you can escape quotes and slashes by doubling them up prior to insertion.
Full example at bottom of link
standard SQL update
# as_json must have escape slashes and quotes doubled
query = """\
UPDATE json_sandbox
SET data = '{}'
WHERE id = 1;
""".format(as_json)
with DBConn(*client.conn_args) as c:
c.cursor.execute(query)
c.connection.commit()
parameterized SQL update
# SQL Driver will do the escaping for you
query = """\
UPDATE json_sandbox
SET data = %s
WHERE id = %s;
"""
with DBConn(*client.conn_args) as c:
c.cursor.execute(query, (as_json, 1))
c.connection.commit()
Invalid JSON SQL
{
"abc": 123,
"quotes": "ain't it great",
"multiLine1": "hello\nworld",
"multiLine3": "hello\r\nuniverse\r\n"
}
Valid JSON SQL
{
"abc": 123,
"quotes": "ain''t it great",
"multiLine1": "hello\\nworld",
"multiLine3": "hello\\r\\nuniverse\\r\\n"
}
Python transform:
# must escape the escape characters, so each slash is doubled
# Some MySQL Python libraries also have an escape() or escape_string() method.
as_json = json.dumps(payload) \
.replace("'", "''") \
.replace('\\', '\\\\')
Full example
import json
import yaml
from DataAccessLayer.mysql_va import get_sql_client, DBConn
client = get_sql_client()
def encode_and_store(payload):
as_json = json.dumps(payload) \
.replace("'", "''") \
.replace('\\', '\\\\')
query = """\
UPDATE json_sandbox
SET data = '{}'
WHERE id = 1;
""".format(as_json)
with DBConn(*client.conn_args) as c:
c.cursor.execute(query)
c.connection.commit()
return
def encode_and_store_2(payload):
as_json = json.dumps(payload)
query = """\
UPDATE json_sandbox
SET data = %s
WHERE id = %s;
"""
with DBConn(*client.conn_args) as c:
c.cursor.execute(query, (as_json, 1))
c.connection.commit()
return
def retrieve_and_decode():
query = """
SELECT * FROM json_sandbox
WHERE id = 1
"""
with DBConn(*client.conn_args) as cnx:
cursor = cnx.dict_cursor
cursor.execute(query)
rows = cursor.fetchall()
as_json = rows[0].get('data')
payload = yaml.safe_load(as_json)
return payload
if __name__ == '__main__':
payload = {
"abc": 123,
"quotes": "ain't it great",
"multiLine1": "hello\nworld",
"multiLine2": """
hello
world
""",
"multiLine3": "hello\r\nuniverse\r\n"
}
encode_and_store(payload)
output_a = retrieve_and_decode()
encode_and_store_2(payload)
output_b = retrieve_and_decode()
print("original: {}".format(payload))
print("method_a: {}".format(output_a))
print("method_b: {}".format(output_b))
print('')
print(output_a['multiLine1'])
print('')
print(output_b['multiLine2'])
print('\nAll Equal?: {}'.format(payload == output_a == output_b))
Python example how to insert raw text:
Create a table in MySQL:
create table penguins(id int primary key auto_increment, msg VARCHAR(4000))
Python code:
#!/usr/bin/env python
import sqlalchemy
from sqlalchemy import text
engine = sqlalchemy.create_engine(
"mysql+mysqlconnector://yourusername:yourpassword#yourhostname.com/your_database")
db = engine.connect()
weird_string = "~!##$%^&*()_+`1234567890-={}|[]\;':\""
sql = text('INSERT INTO penguins (msg) VALUES (:msg)')
insert = db.execute(sql, msg=weird_string)
db.close()
Run it, examine output:
select * from penguins
1 ~!##$%^&*()_+`1234567890-={}|[]\;\':"
None of those characters were interpreted on insert.
Although I also think parameter binding should be used, there is also this:
>>> import MySQLdb
>>> example = r"""I don't like "special" chars ¯\_(ツ)_/¯"""
>>> example
'I don\'t like "special" chars \xc2\xaf\\_(\xe3\x83\x84)_/\xc2\xaf'
>>> MySQLdb.escape_string(example)
'I don\\\'t like \\"special\\" chars \xc2\xaf\\\\_(\xe3\x83\x84)_/\xc2\xaf'
Related
I need to replicate the below function in Python. I included the working PHP code and need help on the Python side as I'm now totally lost.
A RFID card is read and tag transferred to Python over serial. This portion of the code works. WHat I am having trouble with is inserting this string of information as the string of information to be looked up in mySQL
<?php
require 'database.php';
$UIDresult=$_POST["UIDresult"];
$Write="<?php $" . "UIDresult='" . $UIDresult . "'; " . "echo $" . "UIDresult;" . "?>";
file_put_contents('UIDContainer.php',$Write);
$pdo = Database::connect();
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$sql = "SELECT * FROM table_mkaccess where id = ?";
$q = $pdo->prepare($sql);
$q->execute(array($UIDresult));
$data = $q->fetch(PDO::FETCH_ASSOC);
Database::disconnect();
$msg = null;
if (null==$data['name']) {
$msg = "0";
$data['id']=$UIDresult;
} else {
$msg = "1";
}
echo $msg;
?>
Python Code I have tried so far, what am I missing.
import serial
import mysql.connector
rfid = serial.Serial(port = "/dev/ttyUSB0", baudrate=9600)
while True:
if (rfid.in_waiting > 0):
UID = rfid.readline()
UID = UID.decode('Ascii')
mydb = mysql.connector.connect(
host = "localhost",
user = "****",
password = "****",
database = "****")
mycursor = mydb.cursor(buffered=True)
sql = "SELECT * FROM table_mkaccess WHERE id = '%s'"
mycursor.execute(sql,(UIDresult,))
data = mycursor.fetchall()
if data ==0:
print('0')
else:
print('1')
UPDATE:
Now receiving the following error
Traceback (most recent call last): File "/home/base/testing3.py",
line 17, in
mycursor.execute(sql,(UIDresult,))
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an
error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near '47D0D393\r\n'''
at line 1
A few changes:
UID = UID.rstrip() to get rid of the trailing white space (carriage return and newline characters, unless you actually want those characters stored in the database)
sql = "SELECT * FROM table_mkaccess WHERE id = %s" (For prepared statements, you do not want quotes around the %s placeholder. If the supplied actual value is a string, the SQL driver will do "the right thing" with the value.)
mycursor.execute(sql, (UID,)) (Use the correct variable)
You have to add the UID to the query polacholder
mycursor = mydb.cursor(buffered=True)
sql = "SELECT * FROM table_mkaccess WHERE id = %s"
mycursor.execute(sql,(UID,))
data = mycursor.fetchall()
I am using case when in sqlAlchemy as shown below:
abc = "%abc%"
def = "%def%"
proj1 = "%project1%"
proj2 = "%project2%"
case_condition = case([
(text('FPro.status = "ON" and Ab.name.like (''' + abc + ''') and F.project.like (''' + proj1 ''')'''), 'value1'),
(text('FPro.status = "ON" and Ab.name.like (''' + abc + ''') and F.project.like (''' + proj2 + ''')'''), 'value2'),
(text('FPro.status = "OFF" and Ab.name.like (''' + def + ''') and F.project.like (''' + abc + ''')'''), 'value3')]).label (deriver_vals)
query = db.session.query(F)\
.join(FPro, F.id == FPro.f_id)\
.join(Ab, Ab.id == F.ab_id).with_entities(FPro.f_id, case_condition,
F.f_name, F.proj,
FPro.status).subquery()
main_query = db.session.query(Tags).join(query, Tags.tag == query.c.derived_vals).\
with_entities(query.c.f_id.label('f_id'), query.c.derived_vals.label('derived_vals'), Tags.id.label('tag_id')).all()
The above code generates a sql statement like below:
SELECT anon_1.f_id AS f_id, anon_1.derived_vals AS derived_vals, fw_tags.id AS tag_id
FROM fw_tags INNER JOIN (SELECT fpro.f_id AS f_id, CASE WHEN FPro.status = "ON" and Ab.name.like (%%abc%%) and F.project.like (%%proj1%%) THEN %(param_1)s WHEN FPro.status = "ON" and Ab.name.like ( %%abc%%) and Flow.project.like ( %%proj2%%) THEN %(param_2)s WHEN FPro.status = "ON" and Ab.name.like (%%def%%) and F.project.like (%%proj1%%) THEN %(param_3)s END AS derived_vals, F.f_name , F.proj AS project,
FROM F INNER JOIN FPro ON f.id = Fpro.f_id INNER JOIN Ab ON Ab.id = F.ab_id) AS anon_1 ON fw_tags.tag = anon_1.derived_vals]
This is exactly the query I want but I am getting below error while executing the script which contains above code:
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '(%abc%) and F.project.like (%proj1%) THEN 'value1' WHEN FPro' at line 2")
I am guessing the error is with the '%' getting appended, but I am not sure, so any help regarding why this error is occuring or what can be done to prevent the % getting added will be appreciated.
Is there a way to do this without using text or literal()?
In standard SQL "ON" is a delimited/quoted identifier, not a (text) literal, identifying a column, table, or some other schema object. Use single quotes instead: 'ON'. Some DBMS have modes that allow using double quotes for literals, or even attempt to infer the meaning from context, but perhaps it is not a good idea to get into the habit.
In Python 'FPro.status ...''' is the concatenation of 2 string literals, 'FPro.status ...' and '', not a single literal with an escaped single quote in it.
Please do not concatenate or otherwise manually format values to SQL queries, it is error prone as you have found out. It should be obvious from the generated SQL that the values are concatenated as is, without proper quoting, and so produce the incorrect statement. The correct way to pass values to (raw) SQL queries is to use placeholders, or in case of SQLAlchemy, use the SQL Expression Language.
Using placeholders:
abc = "%abc%"
def_ = "%def%"
proj1 = "%project1%"
proj2 = "%project2%"
# Using **raw SQL** fragments
case_condition = case([
(text("FPro.status = 'ON' AND Ab.name LIKE :abc AND F.project LIKE :proj1"), 'value1'),
(text("FPro.status = 'ON' AND Ab.name LIKE :abc AND F.project LIKE :proj2"), 'value2'),
(text("FPro.status = 'OFF' AND Ab.name LIKE :def_ AND F.project LIKE :abc"), 'value3')
])
query = db.session.query(F)\
.join(FPro, F.id == FPro.f_id)\
.join(Ab, Ab.id == F.ab_id)\
.with_entities(
FPro.f_id,
case_condition.label('derived_vals'),
F.f_name,
F.proj,
FPro.status)\
.subquery()
main_query = db.session.query(Tags)\
.join(query, Tags.tag == query.c.derived_vals)\
.with_entities(
query.c.f_id.label('f_id'),
query.c.derived_vals.label('derived_vals'),
Tags.id.label('tag_id'))\
.params(abc=abc, def_=def_, proj1=proj1, proj2=proj2)\
.all()
Using the expression language:
from sqlalchemy import and_
abc = "%abc%"
def_ = "%def%"
proj1 = "%project1%"
proj2 = "%project2%"
# Using SQLAlchemy SQL Expression Language DSL **in Python**
case_condition = case([
(and_(FPro.status == 'ON', Ab.name.like(abc), F.project.like(proj1)), 'value1'),
(and_(FPro.status == 'ON', Ab.name.like(abc), F.project.like(proj2)), 'value2'),
(and_(FPro.status == 'OFF', Ab.name.like(def_), F.project.like(abc)), 'value3')
])
query = db.session.query(F)\
.join(FPro, F.id == FPro.f_id)\
.join(Ab, Ab.id == F.ab_id)\
.with_entities(
FPro.f_id,
case_condition.label('derived_vals'),
F.f_name,
F.proj,
FPro.status)\
.subquery()
main_query = db.session.query(Tags)\
.join(query, Tags.tag == query.c.derived_vals)\
.with_entities(
query.c.f_id.label('f_id'),
query.c.derived_vals.label('derived_vals'),
Tags.id.label('tag_id'))\
.all()
I have a query similar to below:
def connection():
pcon = pg8000.connect(host='host', port=1234, user='user', password='password', database = 'database')
return pcon, pcon.cursor()
pcon, pcur = connection()
query = """ SELECT * FROM db WHERE (db.foo LIKE 'string-%' OR db.foo LIKE 'bar-%')"""
db = pd.read_sql_query(query, pcon)
However when I try to run the code I get:
DatabaseError: '%'' not supported in a quoted string within the query string
I have tried escaping the symbol with \ and an additional % with no luck. How can I get pg8000 to treat this as a wildcard properly?
"In Python, % usually refers to a variable that follows the string. If you want a literal percent sign, then you need to double it. %%"
-- Source
LIKE 'string-%%'
Otherwise, if that doesn't work, PostgreSQL also supports underscores for pattern matching.
'abc' LIKE 'abc' true
'abc' LIKE 'a%' true
'abc' LIKE '_b_' true
But, as mentioned in the comments,
An underscore (_) in pattern stands for (matches) any single character; a percent sign (%) matches any sequence of zero or more characters
According to the source code, though, it would appear the problem is the single quote following the % in your LIKE statement.
if next_c == "%":
in_param_escape = True
else:
raise InterfaceError(
"'%" + next_c + "' not supported in a quoted "
"string within the query string")
So if next_c == "'" instead of next_c == "%", then you would get your error
'%'' not supported in a quoted string within the query string
With a recent version of pg8000 you shouldn't have any problems with a % in a LIKE. For example:
>>> import pg8000.dbapi
>>>
>>> con = pg8000.dbapi.connect(user="postgres", password="cpsnow")
>>> cur = con.cursor()
>>> cur.execute("CREATE TEMPORARY TABLE book (id SERIAL, title TEXT)")
>>> for title in ("Ender's Game", "The Magus"):
... cur.execute("INSERT INTO book (title) VALUES (%s)", [title])
>>>
>>> cur.execute("SELECT * from book WHERE title LIKE 'The %'")
>>> cur.fetchall()
([2, 'The Magus'],)
I am trying to create a statement as follows:
SELECT * FROM table WHERE provider IN ('provider1', 'provider2', ...)
However, I'm having some trouble with the string formatting of it from the Django API. Here's what I have so far:
profile = request.user.get_profile()
providers = profile.provider.values_list('provider', flat=True) # [u'provider1', u'provider2']
providers = tuple[str(item) for item in providers] # ('provider1', 'provider2')
SQL = "SELECT * FROM table WHERE provider IN %s"
args = (providers,)
cursor.execute(sql,args)
DatabaseError
(1241, 'Operand should contain 1 column(s)')
MySQLdb has a method to help with this:
Doc
string_literal(...)
string_literal(obj) -- converts object obj into a SQL string literal.
This means, any special SQL characters are escaped, and it is enclosed
within single quotes. In other words, it performs:
"'%s'" % escape_string(str(obj))
Use connection.string_literal(obj), if you use it at all.
_mysql.string_literal(obj) cannot handle character sets.
Usage
# connection: <_mysql.connection open to 'localhost' at 1008b2420>
str_value = connection.string_literal(tuple(provider))
# '(\'provider1\', \'provider2\')'
SQL = "SELECT * FROM table WHERE provider IN %s"
args = (str_value,)
cursor.execute(sql,args)
Another answer that I don't like particularly, but will work for your apparent use-case:
providers = tuple[str(item) for item in providers] # ('provider1', 'provider2')
# rest of stuff...
SQL = 'SELECT * FROM table WHERE provider IN {}'.format(repr(providers))
cursor.execute(SQL)
You should probably do the string replacement before passing it to the cursor object to execute:
sql = "SELECT * FROM table WHERE provider IN (%s)" % \
(','.join(str(x) for x in providers))
cursor.execute(sql)
So, you have string input for ID's required:
some_vals = '1 3 5 76 5 4 2 5 7 8'.split() # convert to suitable type if required
SomeModel.objects.filter(provider__in=some_vals)
"SELECT * FROM table WHERE provider IN ({0},{1},{2})".format(*args) #where args is list or tuple of arguments.
try this.... should work.
SQL = "SELECT * FROM table WHERE provider IN %s"%(providers)
exec 'cursor.execute("%s")'%(SQL)
I have a JSON object in Python. I am Using Python DB-API and SimpleJson. I am trying to insert the json into a MySQL table.
At moment am getting errors and I believe it is due to the single quotes '' in the JSON Objects.
How can I insert my JSON Object into MySQL using Python?
Here is the error message I get:
error: uncaptured python exception, closing channel
<twitstream.twitasync.TwitterStreamPOST connected at
0x7ff68f91d7e8> (<class '_mysql_exceptions.ProgrammingError'>:
(1064, "You have an error in your SQL syntax; check the
manual that corresponds to your MySQL server version for
the right syntax to use near ''favorited': '0',
'in_reply_to_user_id': '52063869', 'contributors':
'NULL', 'tr' at line 1")
[/usr/lib/python2.5/asyncore.py|read|68]
[/usr/lib/python2.5/asyncore.py|handle_read_event|390]
[/usr/lib/python2.5/asynchat.py|handle_read|137]
[/usr/lib/python2.5/site-packages/twitstream-0.1-py2.5.egg/
twitstream/twitasync.py|found_terminator|55] [twitter.py|callback|26]
[build/bdist.linux-x86_64/egg/MySQLdb/cursors.py|execute|166]
[build/bdist.linux-x86_64/egg/MySQLdb/connections.py|defaulterrorhandler|35])
Another error for reference
error: uncaptured python exception, closing channel
<twitstream.twitasync.TwitterStreamPOST connected at
0x7feb9d52b7e8> (<class '_mysql_exceptions.ProgrammingError'>:
(1064, "You have an error in your SQL syntax; check the manual
that corresponds to your MySQL server version for the right
syntax to use near 'RT #tweetmeme The Best BlackBerry Pearl
Cell Phone Covers http://bit.ly/9WtwUO''' at line 1")
[/usr/lib/python2.5/asyncore.py|read|68]
[/usr/lib/python2.5/asyncore.py|handle_read_event|390]
[/usr/lib/python2.5/asynchat.py|handle_read|137]
[/usr/lib/python2.5/site-packages/twitstream-0.1-
py2.5.egg/twitstream/twitasync.py|found_terminator|55]
[twitter.py|callback|28] [build/bdist.linux-
x86_64/egg/MySQLdb/cursors.py|execute|166] [build/bdist.linux-
x86_64/egg/MySQLdb/connections.py|defaulterrorhandler|35])
Here is a link to the code that I am using http://pastebin.com/q5QSfYLa
#!/usr/bin/env python
try:
import json as simplejson
except ImportError:
import simplejson
import twitstream
import MySQLdb
USER = ''
PASS = ''
USAGE = """%prog"""
conn = MySQLdb.connect(host = "",
user = "",
passwd = "",
db = "")
# Define a function/callable to be called on every status:
def callback(status):
twitdb = conn.cursor ()
twitdb.execute ("INSERT INTO tweets_unprocessed (text, created_at, twitter_id, user_id, user_screen_name, json) VALUES (%s,%s,%s,%s,%s,%s)",(status.get('text'), status.get('created_at'), status.get('id'), status.get('user', {}).get('id'), status.get('user', {}).get('screen_name'), status))
# print status
#print "%s:\t%s\n" % (status.get('user', {}).get('screen_name'), status.get('text'))
if __name__ == '__main__':
# Call a specific API method from the twitstream module:
# stream = twitstream.spritzer(USER, PASS, callback)
twitstream.parser.usage = USAGE
(options, args) = twitstream.parser.parse_args()
if len(args) < 1:
args = ['Blackberry']
stream = twitstream.track(USER, PASS, callback, args, options.debug, engine=options.engine)
# Loop forever on the streaming call:
stream.run()
use json.dumps(json_value) to convert your json object(python object) in a json string that you can insert in a text field in mysql
http://docs.python.org/library/json.html
To expand on the other answers:
Basically you need make sure of two things:
That you have room for the full amount of data that you want to insert in the field that you are trying to place it. Different database field types can fit different amounts of data.
See: MySQL String Datatypes. You probably want the "TEXT" or "BLOB" types.
That you are safely passing the data to database. Some ways of passing data can cause the database to "look" at the data and it will get confused if the data looks like SQL. It's also a security risk. See: SQL Injection
The solution for #1 is to check that the database is designed with correct field type.
The solution for #2 is use parameterized (bound) queries. For instance, instead of:
# Simple, but naive, method.
# Notice that you are passing in 1 large argument to db.execute()
db.execute("INSERT INTO json_col VALUES (" + json_value + ")")
Better, use:
# Correct method. Uses parameter/bind variables.
# Notice that you are passing in 2 arguments to db.execute()
db.execute("INSERT INTO json_col VALUES %s", json_value)
Hope this helps. If so, let me know. :-)
If you are still having a problem, then we will need to examine your syntax more closely.
The most straightforward way to insert a python map into a MySQL JSON field...
python_map = { "foo": "bar", [ "baz", "biz" ] }
sql = "INSERT INTO your_table (json_column_name) VALUES (%s)"
cursor.execute( sql, (json.dumps(python_map),) )
You should be able to insert intyo a text or blob column easily
db.execute("INSERT INTO json_col VALUES %s", json_value)
You need to get a look at the actual SQL string, try something like this:
sqlstr = "INSERT INTO tweets_unprocessed (text, created_at, twitter_id, user_id, user_screen_name, json) VALUES (%s,%s,%s,%s,%s,%s)", (status.get('text'), status.get('created_at'), status.get('id'), status.get('user', {}).get('id'), status.get('user', {}).get('screen_name'), status)
print "about to execute(%s)" % sqlstr
twitdb.execute(sqlstr)
I imagine you are going to find some stray quotes, brackets or parenthesis in there.
#route('/shoes', method='POST')
def createorder():
cursor = db.cursor()
data = request.json
p_id = request.json['product_id']
p_desc = request.json['product_desc']
color = request.json['color']
price = request.json['price']
p_name = request.json['product_name']
q = request.json['quantity']
createDate = datetime.now().isoformat()
print (createDate)
response.content_type = 'application/json'
print(data)
if not data:
abort(400, 'No data received')
sql = "insert into productshoes (product_id, product_desc, color, price, product_name, quantity, createDate) values ('%s', '%s','%s','%d','%s','%d', '%s')" %(p_id, p_desc, color, price, p_name, q, createDate)
print (sql)
try:
# Execute dml and commit changes
cursor.execute(sql,data)
db.commit()
cursor.close()
except:
# Rollback changes
db.rollback()
return dumps(("OK"),default=json_util.default)
One example, how add a JSON file into MySQL using Python. This means that it is necessary to convert the JSON file to sql insert, if there are several JSON objects then it is better to have only one call INSERT than multiple calls, ie for each object to call the function INSERT INTO.
# import Python's JSON lib
import json
# use JSON loads to create a list of records
test_json = json.loads('''
[
{
"COL_ID": "id1",
"COL_INT_VAULE": 7,
"COL_BOOL_VALUE": true,
"COL_FLOAT_VALUE": 3.14159,
"COL_STRING_VAULE": "stackoverflow answer"
},
{
"COL_ID": "id2",
"COL_INT_VAULE": 10,
"COL_BOOL_VALUE": false,
"COL_FLOAT_VALUE": 2.71828,
"COL_STRING_VAULE": "http://stackoverflow.com/"
},
{
"COL_ID": "id3",
"COL_INT_VAULE": 2020,
"COL_BOOL_VALUE": true,
"COL_FLOAT_VALUE": 1.41421,
"COL_STRING_VAULE": "GIRL: Do you drink? PROGRAMMER: No. GIRL: Have Girlfriend? PROGRAMMER: No. GIRL: Then how do you enjoy life? PROGRAMMER: I am Programmer"
}
]
''')
# create a nested list of the records' values
values = [list(x.values()) for x in test_json]
# print(values)
# get the column names
columns = [list(x.keys()) for x in test_json][0]
# value string for the SQL string
values_str = ""
# enumerate over the records' values
for i, record in enumerate(values):
# declare empty list for values
val_list = []
# append each value to a new list of values
for v, val in enumerate(record):
if type(val) == str:
val = "'{}'".format(val.replace("'", "''"))
val_list += [ str(val) ]
# put parenthesis around each record string
values_str += "(" + ', '.join( val_list ) + "),\n"
# remove the last comma and end SQL with a semicolon
values_str = values_str[:-2] + ";"
# concatenate the SQL string
table_name = "json_data"
sql_string = "INSERT INTO %s (%s)\nVALUES\n%s" % (
table_name,
', '.join(columns),
values_str
)
print("\nSQL string:\n\n")
print(sql_string)
output:
SQL string:
INSERT INTO json_data (COL_ID, COL_INT_VAULE, COL_BOOL_VALUE, COL_FLOAT_VALUE, COL_STRING_VAULE)
VALUES
('id1', 7, True, 3.14159, 'stackoverflow answer'),
('id2', 10, False, 2.71828, 'http://stackoverflow.com/'),
('id3', 2020, True, 1.41421, 'GIRL: Do you drink? PROGRAMMER: No. GIRL: Have Girlfriend? PROGRAMMER: No. GIRL: Then how do you enjoy life? PROGRAMMER: I am Programmer.');
The error may be due to an overflow of the size of the field in which you try to insert your json. Without any code, it is hard to help you.
Have you considerate a no-sql database system such as couchdb, which is a document oriented database relying on json format?
Here's a quick tip, if you want to write some inline code, say for a small json value, without import json.
You can escape quotes in SQL by a double quoting, i.e. use '' or "", to enter ' or ".
Sample Python code (not tested):
q = 'INSERT INTO `table`(`db_col`) VALUES ("{k:""some data"";}")'
db_connector.execute(q)