I want to query from Sqlite and contains \n and I expect python converts it to newline but it doesn't. and I also changed \n to \n in the database but still can't be converted.
cursor.execute('''SELECT test FROM table_name ''')
for row in cursor:
self.ui.textEdit.append(row[0])
# or
print row[0]
I also tried unicode(row[0]) and not working. I am surprised there is no an easy solution for this in the web.
Neither SQLite nor Python convert characters in strings (except for \ escapes in a Python string written in the source code).
Newlines work correctly if you handle them correctly:
>>> import sqlite3
>>> db = sqlite3.connect(':memory:')
>>> c = db.cursor()
>>> c.execute('create table t(x)')
>>> c.execute("insert into t values ('x\ny')")
>>> c.execute("insert into t values ('x\\ny')")
>>> c.execute("select * from t")
>>> for row in c:
... print row[0]
...
x
y
x\ny
Related
Trying to pick up some python. I'm quite new to it at the moment.
I created the code below, but it returns an error.
I am able to get it to work when creating a second column and write multiple values to the db but a single value doesn't seem to work. Probably a list, tuple thing, but can not figure out what exactly.
Error:
Traceback (most recent call last):
File "test.py", line 15, in <module>
cursor.executemany("INSERT INTO combination VALUES (?)", combination)
sqlite3.ProgrammingError: Incorrect number of bindings supplied. The current statement uses 1, and there are 2 supplied.
Code:
import sqlite3
conn = sqlite3.connect("combinations.db")
cursor = conn.cursor()
cursor.execute(r"create table if not exists combination (string text)")
combination = []
chars = "abcd"
for char1 in chars:
for char2 in chars:
combination.append((char1+char2))
cursor.executemany("INSERT INTO combination VALUES (?)", combination)
conn.commit()
You missed making the string into a tuple when adding to the list. The argument to executemany expects a list of iterables, so if you pass it a single string 'ab' in the list, it will treat it as a 2-item iterator of a & b - hence the error.
You need to make the string 'ab' into a 1-item tuple like ('ab',). You do this by adding a trailing comma to the expression you're appending:
combination.append((char1+char2,))
Full code:
import sqlite3
conn = sqlite3.connect("combinations.db")
cursor = conn.cursor()
cursor.execute(r"create table if not exists combination (string text)")
combination = []
chars = "abcd"
for char1 in chars:
for char2 in chars:
combination.append((char1+char2,)) # ('ab',) etc.
cursor.executemany("INSERT INTO combination VALUES (?)", combination)
conn.commit()
Error:
pymysql.err.InternalError: (1366, "Incorrect string value: '\\xEF\\xBF\\xBD 20...' for column 'history' at row 1")
I've received a few variations of this as I've tried to tweak my dictionary, always in the history column, the only variations is the characters it tells me are issues.
I can't post the dictionary because it's got sensitive information, but here is the jist:
I started with 200 addresses (including state, zip, etc) that needed
to be validated, normalized and standardized for DB insertion.
I spent a lot of time on google maps validating and standardizing.
I decided to get fancy, and put all the crazy accented letters in the addresses of these world addresses (often copies from google because I don't know how to type and A with an o over it, lol), Singapore to Brazil, everywhere.
I ended up with 120 unique addresses in my dictionary after processing.
Everything works 100% perfectly when INSERTING the data in SQLite and OUTPUTING to a CSV. The issue is exclusively with MySQL and some sneaky un-viewable characters.
Note: I used this to remove the accents after 7 hours of copy/pasting to notepad, encoding it with notepad++ and just trying to processes the data in a way that made it all the correct encoding. I think I did lose the version with the accents and only have this tools output now.
I do not see "\xEF\xBF\xBD 20..." in my dictionary I only see text. Currently I don't even see "20"... those two chars helped me find the previous issues.
Code I can show:
def insert_tables(cursor, assets_final, ips_final):
#Insert Asset data into asset table
field_names_dict = get_asset_field_names(assets_final)
sql_field_names = ",".join(field_names_dict.keys())
for key, row in assets_final.items():
insert_sql = 'INSERT INTO asset(' + sql_field_names + ') VALUES ("' + '","'.join(field_value.replace('"', "'") for field_value in list(row.values())) + '")'
print(insert_sql)
cursor.execute(insert_sql)
#Insert IP data into IP table
field_names_dict = get_ip_field_names(ips_final)
sql_field_names = ",".join(field_names_dict.keys())
for hostname_key, ip_dict in ips_final.items():
for ip_key, ip_row in ip_dict.items():
insert_sql = 'INSERT INTO ip(' + sql_field_names + ') VALUES ("' + '","'.join(field_value.replace('"', "'") for field_value in list(ip_row.values())) + '")'
print(insert_sql)
cursor.execute(insert_sql)
def output_sqlite_db(sqlite_file, assets_final, ips_final):
conn = sqlite3.connect(sqlite_file)
cursor = conn.cursor()
insert_tables(cursor, assets_final, ips_final)
conn.commit()
conn.close()
def output_mysql_db(assets_final, ips_final):
conn = mysql.connect(host=config.mysql_ip, port=config.mysql_port, user=config.mysql_user, password=config.mysql_password, charset="utf8mb4", use_unicode=True)
cursor = conn.cursor()
cursor.execute('USE ' + config.mysql_DB)
insert_tables(cursor, assets_final, ips_final)
conn.commit()
conn.close()
EDIT: Could this have something to do with the fact I'm using Cygwin as my terminal? HA! I added this line and got a different message (now using the accented version again):
cursor.execute('SET NAMES utf8')
Error:
pymysql.err.InternalError: (1366, "Incorrect string value: '\\xC5\\x81A II...' for column 'history' at row 1")
I can shine a bit of light on the messages that you have supplied:
Case 1:
>>> import unicodedata as ucd
>>> s1 = b"\xEF\xBF\xBD"
>>> s1
b'\xef\xbf\xbd'
>>> u1 = s1.decode('utf8')
>>> u1
'\ufffd'
>>> ucd.name(u1)
'REPLACEMENT CHARACTER'
>>>
Looks like you have obtained some bytes encoded in an encoding other than utf8 (e.g. cp1252) then tried bytes.decode(encoding='utf8', errors='strict'). This detected some errors. You then decoded again with errors="replace". This raised no exceptions. However your data has had the error bytes replaced by the replacement character (U+FFFD). Then you encoded your data using str.encodeso that you could write to a file or database. Each replacement characters turns up as 3 hex bytes EF BF BD.
... more to come
Case 2:
>>> s2 = b"\xC5\x81A II"
>>> s2
b'\xc5\x81A II'
>>> u2 = s2.decode('utf8')
>>> u2
'\u0141A II'
>>> ucd.name(u2[0])
'LATIN CAPITAL LETTER L WITH STROKE'
>>>
def quantity():
i = 0
x = 1
file = open("john.txt", "r")
while i < 5000:
for line in file:
c.execute("INSERT INTO test (playerNAME, playerID) VALUES ("+line+", "+str(x)+")")
conn.commit()
x = random.randint(100,10000000000000000)
i += 1
I try to iterate through the John.txt file and insert each value into a table. The first word in the txt file is "abc123". When I run this code there is an error: sqlite3.OperationalError: no such column: abc123
I can get the code to enter the random numbers into playerID but I can't get the txt file query to work...
You need single quotes around the string.
c.execute("INSERT INTO test (playerNAME, playerID) VALUES ('"+line+"', "+str(x)+")")
Otherwise it tries to interpret it as a sql expression and looks for the named column.
More generally you should use parameters or sanitize the incoming data from the file for safety against sql insertion. Even if you trust this particular file. It's a good habit.
c.execute("INSERT INTO test (playerName, playerID) VALUES (?, ?)", (line, x))
Details are here and here is why it's important.
Formatting sql queries via string concatenation is very bad practice.
Variable bindging should always be used:
c.execute("INSERT INTO test (playerNAME, playerID) VALUES (?, ?)", [line, x])
In your case the line probably contains spaces or any punctuation mark.
The sqlite's error string is misleading, though.
I'm querying a database which from the MySQL workbench returns the following value:
Vitória da Conquista
which should be displayed as:
Vitória da Conquista
No matter what I've tried I can't get convert 'Vit\xc3\xb3ria da Conquista' into 'Vitória da Conquista'
#Querying MySQL "world" database
print "====================================="
query = 'select name from city where id=283;'
cursor.execute(query)
cities = cursor.fetchall()
print cities
for city in cities:
cs = str(city)
cs = cs[3:-3].decode('utf-8')
print cs
print cs.decode('utf-8')
print cs.encode('ascii','ignore')
the output of which looks like:
=====================================
[(u'Vit\xc3\xb3ria da Conquista',)]
Vit\xc3\xb3ria da Conquista
Vit\xc3\xb3ria da Conquista
Vit\xc3\xb3ria da Conquista
Well, this actually worked. I'm not sure why however. But I am getting the correct value of Vitória da Conquista. I would like to understand what is happening however.
#Querying MySQL "world" database
query = 'SELECT CONVERT(CAST(Name as BINARY) USING utf8) from city where id = 283;'
cursor.execute(query)
cities = cursor.fetchall()
for tup in cities:
cs=tup[0]
print cs
If the data coming in is in UTF-8 (which looks like it is), use (in Python 2), unicode() to convert it from bytes to a Python Unicode string:
cs = unicode(cs[3:-3], "utf-8")
Basic rule: inside your code, always use Unicode strings. Convert with unicode() input data and with encode() output data.
You are getting unicode strings back, stored in a list of tuples, which is what fetchall does. So you don't need to encode or decode at all. Just try this:
#Querying MySQL "world" database
print "====================================="
query = 'select name from city where id=283;'
cursor.execute(query)
cities = cursor.fetchall()
for tup in cities:
cs = tup[0]
print cs
If this doesn't print right, then you probably have issues with your terminal, as mentioned by #Jarrod Roberson. The only other possibility is that the data was entered into, or is being returned from, the database with the wrong (unexpected) encoding.
I'm trying to print cyrillic chars selected from mysql. Here is my code:
content id DB is cp1251
>>> db = MySQLdb.connect(host="localhost", user="XXX", passwd="XXXX" )
>>> cursor = db.cursor()
>>> cursor.execute("""select id,title,cat,text,tags,date from db1.table1;""")
>>> test=cursor.fetchone()
>>> somevar=test[1]
>>> somevar=somevar.decode('utf8')
>>> print somevar
Result: ?????? ?? ????????
Please guide me how to print this correctly. Thx.
This helped me (got it from here):
db = MySQLdb.connect("localhost", config.db_user, config.db_pwd, config.db_name)
# here's the magic
db.set_character_set("utf8")
dbc = db.cursor()
dbc.execute("SET NAMES utf8;")
dbc.execute("SET CHARACTER SET utf8;")
dbc.execute("SET character_set_connection=utf8;")
# and here goes your SELECT for cyrillic fields
dbc.execute("SELECT id, title, cat, text, tags, date FROM db1.table1;")
# and then you just get the results
test = dbc.fetchone()
somevar = test[1]
print somevar
try this:
somevar = somevar.decode('cp1251')
If that does not help, try to add charset='cp1251' parameter in MySQLdb.connect and there is use_unicode parameter, maybe you should use it to...
all connect parameter you can find here https://github.com/farcepest/MySQLdb1/blob/master/MySQLdb/connections.py
use_unicode
If True, text-like columns are returned as unicode objects
using the connection's character set. Otherwise, text-like
columns are returned as strings. columns are returned as
normal strings. Unicode objects will always be encoded to
the connection's character set regardless of this setting.
charset
If supplied, the connection character set will be changed
to this character set (MySQL-4.1 and newer). This implies
use_unicode=True.