I built a string with a tuple like this:
t = tuple(data)
querysring="INSERT INTO %s VALUES %s "%(table,t)
When I print the string the result is:
INSERT INTO AGENT VALUES ('Bock', 'Fran\\xc3\\xa7ois Bock', 'Individual', 'fb****#mail.com')
But I want something like this:
INSERT INTO AGENT VALUES ('Bock', 'François Bock', 'Individual', 'fb****#mail.com')
It is possible to decode the string ?
I'm using Python2.x but I can use Python3.x
I try this:
querysring=u"INSERT INTO %s VALUES %s "%(table,t)
print(ftfy.fix_text(querysring))
But it's not working
I think your issue is superficial and related to how print displays lists and list items differently. The printed output of a list is in ascii even if the items within the list are correctly encoded in utf-8. First, using chardet library:
from chardet.universaldetector import UniversalDetector
a = ['Bock', 'François Bock']
detector = UniversalDetector()
detector.feed(str(a))
detector.close()
print "Encoding for the str(list): ", detector.result
detector = UniversalDetector()
detector.feed(a[1])
detector.close()
print "Encoding for list[1]: ", detector.result
print "The whole list: ", a
print "Item in list: ", a[1]
Aside from the off-putting printouts, it's possible to still write to the database with the correct encoding with a parameterized query. The last part of the below code writes to a file to confirm that the data encoding is preserved:
import sqlite3
conn = sqlite3.connect(":memory:")
conn.text_factory = str
c = conn.cursor()
c.execute("CREATE TABLE IF NOT EXISTS testing(test1 TEXT, test2 TEXT)")
conn.commit()
my_tuple = 'Bock', 'François Bock'
table = 'testing'
placeholders = ', '.join('?' for item in my_tuple)
query = "INSERT INTO {} VALUES ({})".format(table, placeholders)
c.execute(query, my_tuple)
c.execute("SELECT * FROM testing")
all_data = c.fetchone()
# Check the printouts
print all_data
print all_data[1]
# For good measure, write them to a file
with open('check_output.txt', 'w') as outfile:
outfile.write(', '.join(item for item in all_data))
Related
I tried a few different ways, below but having trouble a) removing the width and b) removing the \n with a comma. I have a txt file like the below and I want to take that information and create a table in sqlite (all using python)
"field",width, type
name, 15, string
revenue, 10, decimal
invoice_date, 10, string
amount, 2, integer
Current python code - trying to read in the file, and get the values to pass in the sql statement below
import os
import pandas as pd
dir_path = os.path.dirname(os.path.realpath(__file__))
file = open(str(dir_path) + '/revenue/revenue_table_specifications.txt','r')
lines = file.readlines()
table = lines[2::]
s = ''.join(str(table).split(','))
x = s.replace("\n", ",").strip()
print(x)
sql I want to pass in
c = sqlite3.connect('rev.db') #connnect to DB
try:
c.execute('''CREATE TABLE
revenue_table (information from txt file,
information from txt file,
....)''')
except sqlite3.OperationalError: #i.e. table exists already
pass
This produces something that will work.
def makesql(filename):
s = []
for row in open(filename):
if row[0] == '"':
continue
parts = row.strip().split(", ")
s.append( f"{parts[0]} {parts[2]}" )
return "CREATE TABLE revenue_table (\n" + ",\n".join(s) + ");"
sql = makesql( 'x.csv' )
print(sql)
c.execute( sql )
I am trying to import txt file into sql, but i have an error:
TypeError: can only concatenate str (not "list") to str
My code:
import psycopg2
con = psycopg2.connect(
host = "",
database="",
user = "",
password = "")
cursor = con.cursor()
with open("pom1.txt") as infile:
for line in infile:
data = line.split()
print(data)
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES"
"(" + data + ");")
cursor.execute(query, *data)
con.commit()
Does anyone have an idea how can i solve it? :)
You don't put the actual values into the parameterized query; you put whatever placeholders are appropriate for your library.
data = line.split()
place_holders = ', '.join("%s" for _ in data) # Assuming %s is correct
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES"
"(" + place_holders + ");")
cursor.execute(query, *data)
cursor.execute takes care of inserting each value where a placeholder occurs, ensure things are properly quoted/escaped/etc.
There are several problems here. First, as the error says, you are trying to concatenate a List (which is data) directly to a string.
Second, you should not use + to concatenate your values and your query.
The doc says:
Warning: Never, never, NEVER use Python string concatenation (+) or string parameters interpolation (%) to pass variables to a SQL query string. Not even at gunpoint.
You should only pass the values to the query via %s.
I'm not sure about the use of * in front of data in cursor.execute(query, *data).
Here is a code that should work, though I have nothing at hand for testing it right now:
import psycopg2
con = psycopg2.connect(
host = "",
database="",
user = "",
password = "")
cursor = con.cursor()
with open("pom1.txt") as infile:
for line in infile:
data = line.split()
print(data)
query = ("INSERT INTO Pomiary_Obwod_90(Znacznik, Pomiar_x, Pomiar_y, Pomiar_z) VALUES (%s, %s, %s, %s);")
cursor.execute(query, data)
con.commit()
I am trying to pass the data from a BLOB column into a user defined function, but it shows up in that function as None. Is this just a quirk, or am I doing somethign wrong?
The BLOB data are jpegs I put into an sqlite3 database using Python 2.7.12. The schema for the table is CREATE TABLE data (md5sum TEXT PRIMARY KEY, data BLOB);
import sqlite3
import hashlib
def p_data(x):
print [x]
return x
def p_md(x):
print [x]
return x
db=sqlite3.connect('tagged.db')
db.text_factory = str
db.create_function('P_DATA', 1, p_data)
db.create_function('P_MD', 1, p_md)
r = db.execute('SELECT P_MD(md5sum),P_DATA(data),data FROM data LIMIT 1')
for i in r:
print
# Don't want to print the thousands of bytes in i[-1]
print i[:1]
print hashlib.md5(i[-1]).hexdigest()
python test.py
[u'3040158ef2c323aaa63da499fc821d77']
[None]
('3040158ef2c323aaa63da499fc821d77', None)
3040158ef2c323aaa63da499fc821d77
Edit
I boiled down my main program into the scripts below to make it easier to share an exact equivalent of what I am running. The test binary data (image) is my avatar image.
Easy initializer for tagged.db
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
PRAGMA writable_schema=ON;
INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)VALUES('table','files','files',0,'CREATE VIRTUAL TABLE "files" USING fts3(fname TEXT, orig_name TEXT, tags TEXT, md5sum TEXT)');
CREATE TABLE 'files_content'(docid INTEGER PRIMARY KEY, 'c0fname', 'c1orig_name', 'c2tags', 'c3md5sum');
CREATE TABLE 'files_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'files_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));
CREATE TABLE data (md5sum TEXT PRIMARY KEY, data BLOB);
CREATE INDEX idx1 on data(md5sum);
PRAGMA writable_schema=OFF;
COMMIT;
Contents of test.py
import sqlite3
import hashlib
def p_data(x):
print 'inside p_data'
print '\t', [x]
return x
def p_md(x):
print 'inside p_md'
print '\t', [x]
return x
db=sqlite3.connect('tagged.db')
db.text_factory = str
db.create_function('P_DATA', 1, p_data)
db.create_function('P_MD', 1, p_md)
fname = '4f880f29b27d5b2c14399512b7155f96.png'
with open(fname, 'rb') as f:
data = f.read()
md = hashlib.md5(data).hexdigest()
db.execute('INSERT INTO files VALUES (?,?,?,?)', ('testname.png', fname, 'testtag', md))
db.execute('INSERT INTO data VALUES (?,?)', (md, data))
# Don't commit so tagged.db will be empty each time test.py is started
r = db.execute('SELECT P_MD(md5sum),P_DATA(data),data FROM data LIMIT 1')
for i in r:
print
print i[:1]
print hashlib.md5(i[-1]).hexdigest()
print
print 'Testing "SELECT typeof(data) FROM data LIMIT 1"'
r = db.execute('SELECT typeof(data) FROM data LIMIT 1')
print r.fetchone()
print
print 'Testing "SELECT p_data("Have some text")'
r = db.execute("SELECT p_data('Have some text')")
v = r.fetchone()
print v
print [str(v[0])]
print
print 'Testing "SELECT p_data(x\'112233\')"'
r = db.execute("SELECT p_data(x'112233')")
v = r.fetchone()
print v
Run python test.py
inside p_md
[u'c337ae2a8ebd84b7e50240d875b2729e']
inside p_data
[None]
('c337ae2a8ebd84b7e50240d875b2729e',)
c337ae2a8ebd84b7e50240d875b2729e
Testing "SELECT typeof(data) FROM data LIMIT 1"
('text',)
Testing "SELECT p_data("Have some text")
inside p_data
[u'Have some text']
('Have some text',)
['Have some text']
Testing "SELECT p_data(x'112233')"
inside p_data
[<read-write buffer ptr 0x7f31080cad18, size 3 at 0x7f31080cacd8>]
(<read-write buffer ptr 0x7f31080cad18, size 3 at 0x7f31080cacd8>,)
['\x11"3']
In test.py you'll find that type(data) is str, meaning it's inserted as text and likely getting confused about encoding when you read it back out.
The solution is to insert an sqlite3.Binary instead:
db.execute('INSERT INTO data VALUES (?,?)', (md, sqlite3.Binary(data)))
Let's say i have a python list of customer id like this:
id = ('12','14','15','11',.......)
the array has 1000 values in it, and i need to insert the customer name to a table based on the ids from the list above.
my code is like:
ids = ",".join(id)
sql = "insert into cust_table(name)values(names)where cust_id IN('ids')"
cursor.execute(sql)
after running the code, i get nothing inserted to the table. What mistake do i have?
Please help :(
You need to format the string.
ids = ",".join(id)
sql = "insert into cust_table(name)values(names)where cust_id IN('{ids}')"
cursor.execute(sql.format(ids= ids))
Simply writing the name of a variable into a string doesn't magically make its contents appear in the string.
>>> p = 'some part'
>>> s = 'replace p of a string'
>>> s
'replace p of a string'
>>> s = 'replace %s of a string' % p
>>> s
'replace some part of a string'
>>> s = 'replace {} of a string'.format(p)
>>> s
'replace some part of a string'
In your case this would mean:
>>> sql = "insert into cust_table (name) values (names) where cust_id IN ('%s')"
>>> ids = ", ".join(id)
>>> cursor.execute(sql % ids)
although I strongly suspect that you have a similar problem with names.
In order to avoid possible sql injection problems, it would be preferable to use a "parameterized statement". This would look something like:
>>> sql = 'insert into ... where cust_id IN %s'
>>> cursor.execute(sql, (id,))
Some database connectors for python are capable of this, but yours probably isn't.
A workaround might be something like
>>> params = ', '.join(['%s']*len(id))
>>> sql = 'insert into ... where cust_id IN (%s)' % params
>>> cursor.execute(sql, id)
How to convert
cursor.execute("SELECT strftime('%m.%d.%Y %H:%M:%S', timestamp, 'localtime'), temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
for row in results[:-1]:
row=results[-1]
rowstr="['{0}',{1}]\n".format(str(row[0]),str(row[1]))
temp_chart_table+=rowstr
result
['01.15.2015 21:38:52',21.812]
into dictionary output in form of:
[{timestamp:'01.15.2015 21:38:52',temp:21.812}]
Edit
This is fetchone sample I currenyly use and it works fine:
def get_avg():
conn=sqlite3.connect(dbname)
curs=conn.cursor()
curs.execute("SELECT ROUND(avg(temp), 2.2) FROM data WHERE timestamp>datetime('now','-1 hour') AND timestamp<=datetime('now')")
rowavg=curs.fetchone()
#print rowavg
#rowstrmin=format(str(rowavg[0]))
#return rowstrmin
**d = [{"avg":rowavg[0]}]**
return d
conn.close()
#print get_avg()
schema = {"avg": ("number", "avg")}
data = get_avg()
# Loading it into gviz_api.DataTable
data_table = gviz_api.DataTable(schema)
data_table.LoadData(data)
json = data_table.ToJSon()
#print results
#print "Content-type: application/json\n\n"
print "Content-type: application/json"
print
print json
Then I make jQuery call and pass it into javascript and found help for that in here
ajax json query directly to python generated html gets undefined
As I can see you are using format to write in the form of a string.
Note from the docs
it is not possible to use { and } as fill char while using the str.format() method
To make it look like a dictionary you can do
"[{timestamp:'%s',temp:%s}]\n"%(str(row[0]),str(row[1]))
But if you want to make it a dictionary then you will have to do
row_dic = [{'timestamp':row[0],'temp':row[1]}]
Try this instead:
cursor.execute("SELECT strftime('%m.%d.%Y %H:%M:%S', timestamp, 'localtime'), temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
temp_chart_table = []
for row in results:
temp_chart_table.append({'timestamp': row[0], 'temp': row[1]})
In most of the python database adapters you can use a DictCursor to retrieve records using an interface similar to the Python dictionaries instead of the tuples.
Using psycopg2:
>>> dict_cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
>>> dict_cur.execute("INSERT INTO test (num, data) VALUES(%s, %s)",
... (100, "abc'def"))
>>> dict_cur.execute("SELECT * FROM test")
>>> rec = dict_cur.fetchone()
>>> rec['id']
1
>>> rec['num']
100
>>> rec['data']
"abc'def"
Using MySQLdb:
>>> import MySQLdb
>>> import MySQLdb.cursors
>>> myDb = MySQLdb.connect(user='andy47', passwd='password', db='db_name', cursorclass=MySQLdb.cursors.DictCursor)
>>> myCurs = myDb.cursor()
>>> myCurs.execute("SELECT columna, columnb FROM tablea")
>>> firstRow = myCurs.fetchone()
{'columna':'first value', 'columnb':'second value'}
def stuffToDict(stuff):
return {"timestamp":stuff[0],"temp":stuff[1]}
That would be a dictionary. The sample output you showed is a list of dictionaries, which can be achieved by putting square brackets around the dictionary. I don't know why you'd want that, though. Also, because of the missing quotes, it wasn't legal python syntax.
Use MySQLdb's cursor library.
import MySQLdb
import MySQLdb.cursors
conn = MySQLdb.connect(host=db_host, user=db_user, passwd=db_passwd, db=db_schema, port=db_port, cursorclass=MySQLdb.cursors.DictCursor)
cursor = conn.cursor()
cursor.execute("SELECT timestamp, localtime, temp FROM data WHERE timestamp>datetime('now','-1 hours')")
# fetch all or one we'll go for all.
results = cursor.fetchall()
Then you have access to the results as a dictionary:
>>> results['timestamp']
14146587
>>> results['localtime']
20:08:07
>>> results['temp']
temp_variable_whatever