We Keep Coding

How to insert Hindi language to mysql?

I want to insert Hindi language sentences to mysql database.
But I encountered a problem: Hindi language sentences that are inserted into mysql database has become garbled.
I have set the encoding format to UTF-8, then my code is as follows.
Thanks a lot!
#coding = utf-8
import MySQLdb
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
dbs = MySQLdb.connect(host='x.x.x.x', user='x', passwd = 'x', db='x',port=x)
cursor = dbs.cursor()
with open('hindi.wiki.set','r') as file:
count = 1
for line in file.readlines():
if count == 5:
break
sql = """insert into `lab_ime_test_set_2` (id_, type_, lang_, text_, anno_) values(%s, %s, %s,'%s', %s)""" % ("null", "'wiki'", "'hindi'", MySQLdb.escape_string(line.strip()), "'not_anno'")
try:
cursor.execute(sql)
dbs.commit()
except Exception as eh:
print("error")
print("total count", count)
cursor.close()
dbs.close()
since the sql can bu run in navicat for mysql and the hindi language can be shown correctly.
But when I run this code, the sentences can be inserted in mysql database as well, but can't be shown correctly.
such as "संतरे के जायके वाले मूल टैंग को 1957 में जनरल फूडà¥à¤¸ कॉरपोरेशन के लिठविलियम à¤"

Psycopg2 execute tuple index out of range

I am very new to python and this is my first programming language. This is my first shot at using SQL as well as psycopg2. Any "dumb" advice is much appreciated!
I am not sure what the problem is. My research tells me im feeding too few or too many arguments to the cursor.execute(INSERT... but i have tried a number of different counts and can't seem to get any working correctly. From my point of view the cursor.execute(CREATE... creates a table with 6 columns and i am passing 6 args to it.
from lxml import html # Used to parse XML
import requests #used to service API request
itemtypeid1 = 34
itemtypeid2 = 35
regionid = 10000002
webpage = requests.get('http://api.eve-central.com/api/marketstat?typeid=%i&typeid=%i&regionlimit=%i' % (
itemtypeid1, itemtypeid2, regionid))
if webpage.status_code == 200:
data = html.fromstring(webpage.content)
for item in data.iter('type'):
buy_dict = {node.tag: node.text for node in item.xpath("buy/*")}
sell_dict = {node.tag: node.text for node in item.xpath("sell/*")}
#Variables for output
itemid = (item.get("id"))
buymin = buy_dict['min']
buymax = buy_dict['max']
buymedian = buy_dict['median']
buyvolume = buy_dict['volume']
buyaverage = buy_dict['avg']
#Fail if api webpage unavaliable
else:
print "Webpage unavaliable"
Webpage.raise_for_status()
#############################################################################
import psycopg2
connection = psycopg2.connect(database='evemarketdata', user='postgres', password='black3car')
#open a cursor to perform DB operations
cursor = connection.cursor()
#create new table
cursor.execute("CREATE TABLE arkonor (itemid integer primary key, min integer, max integer, median integer, volume integer, average integer);")
#Insert row data into DB table
cursor.execute("""INSERT INTO arkonor (typeid, min, max, median, volume, average)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)""",
('itemid', 'buymin', 'buymax', 'buymedian', 'buyvolume', 'buyaverage'))
#Commits all changes does with cursor
#connection.commit()
results in
Traceback (most recent call last):
File "E:\Eve Spreadsheets\Python\PostgreConnect.py", line 49, in <module>
('itemid', 'buymin', 'buymax', 'buymedian', 'buyvolume', 'buyaverage'))
IndexError: tuple index out of range

You have 8 parameters in the query but provided only 6 fields in the tuple. The code should be:
#Insert row data into DB table
cursor.execute("""INSERT INTO arkonor (typeid, min, max, median, volume, average)
VALUES (%s, %s, %s, %s, %s, %s)""",
('itemid', 'buymin', 'buymax', 'buymedian', 'buyvolume', 'buyaverage'))

How to use like pattern matching with PostgreSQL and Python with multiple percentage (%) symbols?

I am trying to pattern match with the LIKE LOWER('% %') command however I think the fact that I am using a python variable with %s is mucking it up. I can't seem to find any escape characters for the percentage symbol and my program gives me no errors. Is this the problem or is there something else I'm missing. It does work if I just run LIKE %s however I need to be able to search like not equals.
# Ask for the database connection, and get the cursor set up
conn = database_connect()
if(conn is None):
return ERROR_CODE
cur = conn.cursor()
print("search_term: ", search_term)
try:
# Select the bays that match (or are similar) to the search term
sql = """SELECT fp.name AS "Name", fp.size AS "Size", COUNT(*) AS "Number of Fish"
FROM FishPond fp JOIN Fish f ON (fp.pondID = f.livesAt)
WHERE LOWER(fp.name) LIKE LOWER('%%s%') OR LOWER(fp.size) LIKE LOWER('%%s%')
GROUP BY fp.name, fp.size"""
cur.execute(sql, (search_term, ))
rows = cur.fetchall()
cur.close() # Close the cursor
conn.close() # Close the connection to the db
return rows
except:
# If there were any errors, return a NULL row printing an error to the debug
print("Error with Database - Unable to search pond")
cur.close() # Close the cursor
conn.close() # Close the connection to the db
return None

Instead of embedding the ampersands in the query string, you could wrap the search term string in ampersands, and then pass that to cursor.execute():
sql = 'SELECT * from FishPond fp WHERE LOWER(fp.name) LIKE LOWER(%s)'
search_term = 'xyz'
like_pattern = '%{}%'.format(search_term)
cur.execute(sql, (like_pattern,))
The query is simplified for the purpose of example.
This is more flexible because the calling code can pass any valid LIKE pattern to the query.
BTW: In Postgresql you can use ILIKE for case insensitive pattern matching, so the example query could be written as this:
sql = 'SELECT * from FishPond fp WHERE fp.name ILIKE %s'
As noted in the documentation ILIKE is a Postgresql extension, not standard SQL.

You can escape % with another %
>>> test = 'test'
>>> a = 'LIKE %%s%'
>>> a % test
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: incomplete format
>>>
>>> a = 'LIKE %%%s%%'
>>> a % test
'LIKE %test%'
P.S. you also have two placeholders, but you are passing only one argument in execute

mysqldb programming Error

I'm getting an error trying to insert into a table using MySQLdb and can not figure it out. I looked up how to catch the error but when I add the code to trap it, I don't get any additional info when I tried using raise e instead of print e and when I try print e I get an error. I need some help getting the error to give me more detail. I'll include the whole code in case someone is awesome enough to see where I went wrong in the first place. Then I don't even care about the error trapping. Ok, yes I do because I am sure I will need it again soon. lol. I actually have 8 columns I need to update. I'm trying to use a process of elimination and only trying a couple at a time. The ones like UPC, Name, and a couple others insert fine but Modl, Path, and Desc are causing errors. Any help with either would be appreciated and thanks.
# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
from __future__ import print_function
from datetime import date, datetime, timedelta
import MySQLdb
#from scrapy.extensions import DropItem
#from bff.items import ItemInfo
class mySQLPipeline(object):
def process_item(self, item, spider):
Product = item['ProdName']
Path = item['ProdPath']
UPC = item['ProdUPC']
Modl = item['ProdModel']
Desc = item['ProdDesc']
Price = item['ProdPrice']
Stock = item['InStock']
#Ships = item['Ships']
Name = item['StoreName']
#Not Failing during insert Product, Price, Stock, Name
#FAILing during insert Modl, Path, Desc,
db = MySQLdb.connect(user='****', passwd='****',
host='127.0.0.1',
port=****,
db='****')
cursor = db.cursor()
# add_Product = ("INSERT INTO walmart_products (ProdName, StoreName) VALUES (%s, %s,)", Product, Name,)
# add_Product = ("INSERT INTO walmart_products, (ProdName)"
# "VALUES (%s)", (Name))
# "VALUES (%(Name)s)")
add_Product = ("INSERT INTO walmart_products "
"(InStock, StoreName) "
"VALUES (%s, %s)")
#item['Ships'],
data_Product = (Stock, Name)
#Add new product
#try:
cursor.execute(add_Product, data_Product)
#except MySQLdb.IntegrityError, e:
#print e
# handle a specific error condition
#except MySQLdb.Error, e:
#raise e
# handle a generic error condition
#except MySQLdb.Warning, e:
#print e
# handle warnings, if the cursor you're using raises them
#except MySQLdb.ProgrammingError, e:
#print e
# Make sure data is committed to the database
db.commit()
cursor.close()
db.close()
return item

I figured out what was happening. It was actually a couple of things. One a couple of them I had the column set to varchar(200) and was getting over 2400 chars so increasing it fixed that. On the URL on I wasn't escaping it and that was causing the errors. Thanks again. – Nick just now edit

Chinese character doesn't support for Sqlite3 through CGI script

inserting Chinese character into Sqlite3 through Cgi script is not working for me. I can insert and select Chinese character from same database using Query browser tool but when I use python script for this, it's show error. This is the query i have used for create database
CREATE TABLE registrations (
m_username VARCHAR PRIMARY KEY
COLLATE 'BINARY',
m_identity VARCHAR,
m_updatetime DATETIME
);
and then this is the cgi script i have used for update and select values form the database
#! /Python26/python
dbFile = 'D:/sqlite/registrations'
import cgi
import sqlite3
import xml.sax.saxutils
query = cgi.parse()
db = sqlite3.connect(dbFile)
user = query.get('username', [None])[0]
identity = query.get('identity', [None])[0]
friends = query.get('friends', [])
print 'Content-type: text/plain\n\n<?xml version="1.0" encoding="utf-8"?>\n'
print "<result>"
if user:
try:
c = db.cursor()
c.execute("insert or replace into registrations values (?, ?, datetime('now'))", (user, identity))
print "\t<update>true</update>"
except:
print '\t<update>false</update>'
for f in friends:
print "\t<friend>\n\t\t<user>%s</user>" % (xml.sax.saxutils.escape(f), )
c = db.cursor()
c.execute("select m_username, m_identity from registrations where m_username = ? and m_updatetime > datetime('now', '-1 hour')", (f, ))
for result in c.fetchall():
eachIdent = result[1]
if not eachIdent:
eachIdent = ""
print "\t\t<identity>%s</identity>" % (xml.sax.saxutils.escape(eachIdent), )
if f != result[0]:
print "\t\t<registered>%s</registered>" % (xml.sax.saxutils.escape(result[0]), )
print "\t</friend>"
db.commit()
print "</result>"
I think, i need to set CHARSET as UTF-8 something, but I don't know how to do it. i was googled, but couldn't find good way to solve this issue. kindly some one help me please.

I have done this through the client side. I just used EncodeBase64 and encoded the chinese data and send to the db. I think, this one is not straight way. but I couldn't find any other way.