We Keep Coding

How to store and query hex values in mysqldb

I want to use a thermal printer with raspberry pi. I want to receive the printer vendor id and product id from mysql database. My columns are of type varchar.
My code is
import MySQLdb
from escpos.printer import Usb
db= MySQLdb.connect(host=HOST, port=PORT,user=USER, passwd=PASSWORD, db=database)
cursor = db.cursor()
sql = ("select * from printerdetails")
cursor.execute(sql)
result = cursor.fetchall()
db.close()
for row in result:
printer_vendor_id = row[2]
printer_product_id = row[3]
input_end_point = row[4]
output_end_point = row[5]
print printer_vendor_id,printer_product_id,input_end_point,output_end_point
Printer = Usb(printer_vendor_id,printer_product_id,0,input_end_point,output_end_point)
Printer.text("Hello World")
Printer.cut()
but it doesnot work. the id's are string. print command shows 0x154f 0x0517 0x82 0x02.in my case
Printer = Usb(0x154f,0x0517,0,0x82,0x02)
works fine.How could I store the same id's to the database and use them to configure the printer

Your problem is that your call to Usb is expecting integers, which works if you call it like this
Printer = Usb(0x154f,0x0517,0,0x82,0x02)
but your database call is returning tuples of hexadecimal values stored as strings. So you need to convert those strings to integers, like this:
for row in result:
printer_vendor_id = int(row[2],16)
printer_product_id = int(row[3],16)
input_end_point = int(row[4],16)
output_end_point = int(row[5],16)
Now if you do
print printer_vendor_id,printer_product_id,input_end_point,output_end_point
you will get
(5455, 1303, 130, 2)
which might look wrong, but isn't, which you can check by asking for the integers to be shown in hex format:
print ','.join('0x{0:04x}'.format(i) for i in (printer_vendor_id,printer_product_id,input_end_point,output_end_point))
0x154f,0x0517,0x0082,0x0002
I should point out that this only works because your database table contains only one row. for row in result loops through all of the rows in your table, but there happens to be only one, which is okay. If there were more, your code would always get the last row of the table, because it doesn't check the identifier of the row and so will repeatedly assign values to the same variables until it runs out of data.
The way to fix that is to put a where clause in your SQL select statement. Something like
"select * from printerdetails where id = '{0}'".format(printer_id)
Now, because I don't know what your database table looks like, the column name id is almost certainly wrong. And very likely the datatype also: it might very well not be a string.

Summing database column in python

I have recently encountered the problem of adding the elements of a database column. Here is the following code:
import sqlite3
con = sqlite3.connect("values.db")
cur = con.cursor()
cur.execute('SELECT objects FROM data WHERE firm = "sony"')
As you can see, I connect to the database (sql) and I tell to Python to select the column "objects".
The problem is that I do not know the appropriate command for summing the selected objects.
Any ideas/ advices are highly reccomended.
Thank you in advance!!

If you can, have the database do the sum, as that reduces data transfer and lets the database do what it's good at.
cur.execute("SELECT sum(objects) FROM data WHERE firm = 'sony'")
or, if you're really just looking for the total count of objects.
cur.execute("SELECT count(objects) FROM data WHERE firm = 'sony'")
either way, your result is simply:
count = cur.fetchall()[0][0]

Try the following line:
print sum([ row[0] for row in cur.fetchall()])
If you want the items instead adding them together:
print ([ row[0] for row in cur.fetchall()])

Check if row exists in SQLite with Python

I want to know if a row exists already in one of my tables, in this case coll. In order to do this I played around with SQLite in the shell a little and stumbled upon SELECT EXISTS(SELECT 1 FROM coll WHERE ceeb="1234"). In SQLite this works perfectly and it returns either a 0 or a 1-- which is exactly what I wanted. So, with code in hand, I wrote up a quick Python script to see if I could get this to work for me before sticking it into my program. This is what I came up with:
import sqlite3
conn = sqlite3.connect('stu.db')
c = conn.cursor()
sceeb = int(raw_input(":> "))
ceeb_exists = c.execute('SELECT EXISTS(SELECT 1 FROM coll WHERE ceeb="%d" LIMIT 1)' % sceeb)
print ceeb_exists
Instead of assigning ceeb_existsa 1 or a 0 it gives me an output that looks like <sqlite3.Cursor object at 0x01DF6860>. What am I doing wrong here?

The execution of a query always results in 0 or more rows. You'd need to fetch those rows; a SELECT EXISTS query results in 1 row, so you'd need to fetch that row.
Rows always consist of 1 or more columns, here you get one, so you could use tuple assignment (note the , comma after ceeb_exists):
c.execute('SELECT EXISTS(SELECT 1 FROM coll WHERE ceeb="%d" LIMIT 1)' % sceeb)
ceeb_exists, = c.fetchone()
However, using an EXISTS query is a bit redundant here; you could just test if there is any row returned. You should also use query parameters to avoid a SQL injection attack (you are asking a user to give you the ceeb value, so that is easily hijacked):
c.execute('SELECT 1 FROM coll WHERE ceeb=? LIMIT 1', (sceeb,))
ceeb_exists = c.fetchone() is not None
cursor.fetchone() returns None if there is no row available to fetch, the is not None test turns that into True or False.

.executes() returns a cursor object as you can see.
In order to print the results of the query you need to iterate over it:
for result in exists:
print result

SQL multiple inserts with Python

UPDATE
After passing execute() a list of rows as per Nathan's suggestion, below, the code executes further but still gets stuck on the execute function. The error message reads:
query = query % db.literal(args)
TypeError: not all arguments converted during string formatting
So it still isn't working. Does anybody know why there is a type error now?
END UPDATE
I have a large mailing list in .xls format. I am using python with xlrd to retrieve the name and email from the xls file into two lists. Now I want to put each name and email into a mysql database. I'm using MySQLdb for this part. Obviously I don't want to do an insert statement for every list item.
Here's what I have so far.
from xlrd import open_workbook, cellname
import MySQLdb
dbname = 'h4h'
host = 'localhost'
pwd = 'P#ssw0rd'
user = 'root'
book = open_workbook('h4hlist.xls')
sheet = book.sheet_by_index(0)
mailing_list = {}
name_list = []
email_list = []
for row in range(sheet.nrows):
"""name is in the 0th col. email is the 4th col."""
name = sheet.cell(row, 0).value
email = sheet.cell(row, 4).value
if name and email:
mailing_list[name] = email
for n, e in sorted(mailing_list.iteritems()):
name_list.append(n)
email_list.append(e)
db = MySQLdb.connect(host=host, user=user, db=dbname, passwd=pwd)
cursor = db.cursor()
cursor.execute("""INSERT INTO mailing_list (name,email) VALUES (%s,%s)""",
(name_list, email_list))
The problem when the cursor executes. This is the error: _mysql_exceptions.OperationalError: (1241, 'Operand should contain 1 column(s)') I tried putting my query into a var initially, but then it just barfed up a message about passing a tuple to execute().
What am I doing wrong? Is this even possible?
The list is huge and I definitely can't afford to put the insert into a loop. I looked at using LOAD DATA INFILE, but I really don't understand how to format the file or the query and my eyes bleed when I have to read MySQL docs. I know I could probably use some online xls to mysql converter, but this is a learning exercise for me as well. Is there a better way?

You need to give executemany() a list of rows. You don't need break the name and email out into separate lists, just create one list with both of the values in it.
rows = []
for row in range(sheet.nrows):
"""name is in the 0th col. email is the 4th col."""
name = sheet.cell(row, 0).value
email = sheet.cell(row, 4).value
rows.append((name, email))
db = MySQLdb.connect(host=host, user=user, db=dbname, passwd=pwd)
cursor = db.cursor()
cursor.executemany("""INSERT INTO mailing_list (name,email) VALUES (%s,%s)""", rows)
Update: as #JonClements mentions, it should be executemany() not execute().

To fix TypeError: not all arguments converted during string formatting - you need to use the cursor.executemany(...) method, as this accepts an iterable of tuples (more than one row), while cursor.execute(...) expects the parameter to be a single row value.
After the command is executed, you need to ensure that the transaction is committed to make the changes active in the database by using db.commit().

If you are interested in high-performance of the code, this answer may be better.
Compare to excutemany method, the below execute will much faster:
INSERT INTO mailing_list (name,email) VALUES ('Jim','jim#yahoo.com'),('Lucy','Lucy#gmail.com')
You can easily modify the answer from #Nathan Villaescusa and get the new code.
cursor.execute("""INSERT INTO mailing_list (name,email) VALUES (%s)""".format(",".join(str(i) for i in rows))
here is my own test result:
excutemany:10000 runs takes 220 seconds
execute:10000 runs takes 12 seconds.
The speed difference will be about 15 times.

Taking up the idea of #PengjuZhao, it should work to simply add one single placeholder for all values to be passed. The difference to #PengjuZhao's answer is that the values are passed as a second parameter to the execute() function, which should be injection attack safe because this is only evalutated during runtime (in contrast to ".format()").
cursor.execute("""INSERT INTO mailing_list (name,email) VALUES (%s)""", ",".join(str(i) for i in rows))
Only if this does not work properly, try the approach below.
####
#PengjuZhao's answer shows that executemany() has either a strong Python overhead or it uses multiple execute() statements where this is not needed, elsewise executemany() would not be so much slower than a single execute() statement.
Here is a function that puts NathanVillaescusa's and #PengjuZhao's answers in a single execute() approach.
The solution builds a dynamic number of placeholders to be added to the sql statement. It is a manually built execute() statement with multiple placeholders of "%s", which likely outperforms the executemany() statement.
For example, at 2 columns, inserting 100 rows:
execute(): 200 times "%s" (= dependent from the number of the rows)
executemany(): just 2 times "%s" (= independent from the number of the rows).
There is a chance that this solution has the high speed of #PengjuZhao's answer without risking injection attacks.
Prepare parameters of the function:
You will store your values in 1-dimensional numpy arrays arr_name and arr_email which are then converted in a list of concatenated values, row by row. Alternatively, you use the approach of #NathanVillaescusa.
from itertools import chain
listAllValues = list(chain([
arr_name.reshape(-1,1), arr_email.reshape(-1,1)
]))
column_names = 'name, email'
table_name = 'mailing_list'
Get sql query with placeholders:
The numRows = int((len(listAllValues))/numColumns) simply avoids passing the number of rows. If you insert 6 values in listAllValues at 2 columns this would make 6/2 = 3 rows then, obviously.
def getSqlInsertMultipleRowsInSqlTable(table_name, column_names, listAllValues):
numColumns = len(column_names.split(","))
numRows = int((len(listAllValues))/numColumns)
placeholdersPerRow = "("+', '.join(['%s'] * numColumns)+")"
placeholders = ', '.join([placeholdersPerRow] * numRows)
sqlInsertMultipleRowsInSqlTable = "insert into `{table}` ({columns}) values {values};".format(table=table_name, columns=column_names, values=placeholders)
return sqlInsertMultipleRowsInSqlTable
strSqlQuery = getSqlInsertMultipleRowsInSqlTable(table_name, column_names, listAllValues)
Execute strSqlQuery
Final step:
db = MySQLdb.connect(host=host, user=user, db=dbname, passwd=pwd)
cursor = db.cursor()
cursor.execute(strSqlQuery, listAllValues)
This solution is hopefully without the risk of injection attacks as in #PengjuZhao's answer since it fills the sql statement only with placeholders instead of values. The values are only passed separately in listAllValues at this point here, where strSqlQuery has only placeholders instead of values:
cursor.execute(strSqlQuery, listAllValues)
The execute() statement gets the sql statement with placeholders %s and the list of values in two separate parameters, as it is done in #NathanVillaescusa's answer. I am still not sure whether this avoids injection attacks. It is my understanding that injection attacks can only occur if the values are put directly in the sql statement, please comment if I am wrong.

get raw decimal value from mysqldb query

I am executing a mysql query in python using the MySQLdb package. The code looks something like this:
c=db.cursor()
c.execute("""select * from table""")
output = []
for row in c:
output.append(row[4])
where row[4] contains a decimal value that I want to store in the output list.
The problem is every value that I am getting looks like this: Decimal('XX.XX') where all I want in the output list is XX.XX. At the end of the script, my output list looks like this:
[Decimal('10.02'), Decimal('20.24'), ...]
But I need it to just contain the numbers, like this:
[10.02, 20.24, ...]
How do I do that?
Thanks!

You can either convert a Decimal object to a string:
cursor = db.cursor()
cursor.execute("""select * from table""")
output = []
for row in cursor:
output.append(str(row[4]))
Or to a float:
cursor = db.cursor()
cursor.execute("""select * from table""")
output = []
for row in cursor:
output.append(float(row[4]))
Converting it to a float will cause it to lose its full precision, so a value like 20.24 will become 20.239999999999998.
Also, casting it to a float will raise an Exception if the value is None. To avoid that, you can use a helper function like this:
def convert_mysql_decimal_to_float(decimal_object):
if (decimal_object == None):
return None
else:
return float(decimal_object)
cell_value = convert_mysql_decimal_to_float(row[4])

If you are trying to write a generic code that operates on the column output, the above solution won't workout. In that case we can write our SELECT query in a way that the column in returned as String and we just get the value of what we want.
The query can be framed in below way,
SELECT
CAST(COL1 AS CHAR) AS COL1,
CAST(COL2 AS CHAR) AS COL2,
.
.
.
FROM TABLE;

Use float():
output.append(float(row[4]))
But float() can result in something like:
In [184]: float(Decimal('10.02'))
Out[184]: 10.02
In [185]: float(Decimal('20.24'))
Out[185]: 20.239999999999998

To avoid losing precision, you can use Python's decimal module.
from decimal import Decimal
c=db.cursor()
c.execute("""select * from table""")
output = []
for row in c:
row_data = []
for data in row:
if type(data) is Decimal:
row_data.append(float(data))
else:
row_data.append(str(dat))
output.append(row_data)