I am trying to save hash_value in mysql database using python. I have obtained hash value hash = imagehash.dhash(Image.open('temp_face.jpg')) but after the execution of insert query cursor.execute("INSERT INTO image(hash,name,photo) VALUES(%d,%s,%s )", (hash,name, binary_image))it gives me error "Python 'imagehash' cannot be converted to a MySQL type".
| Field | Type | Null | Key | Default | Extra |
+---------+-------------+------+-----+-------------------+-------------------+
| hash | binary(32) | NO | PRI | NULL | |
| name | varchar(25) | NO | | NULL | |
| photo | blob | NO | | NULL | |
| arrival | datetime | NO | | CURRENT_TIMESTAMP | DEFAULT_GENERATED |
+---------+-------------+------+-----+-------------------+-------------------+
So what can be done to store the value or is there any other way to do the same task?
I have two tables
front_employee (Employee model in Django)
+-----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| order | int(11) | NO | | NULL | |
| name | varchar(100) | YES | | NULL | |
| position | varchar(100) | YES | | NULL | |
| description | longtext | YES | | NULL | |
| employee_img_id | int(11) | NO | MUL | NULL | |
| language_id | int(11) | NO | MUL | NULL | |
+-----------------+--------------+------+-----+---------+----------------+
And front_employeepicture (EmployeePicture in Django)
+-------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| order | int(11) | NO | | NULL | |
| img | varchar(100) | YES | | NULL | |
+-------+--------------+------+-----+---------+----------------+
I would like to perform this query:
SELECT a.id, a.name, b.img
FROM front_employee a
INNER JOIN front_employeepicture b
ON a.employee_img_id = b.id
For now I have
context['employee'] = Employee.objects.all().order_by('order')
And I tried something like
context['employee'] = Employee.objects.select_related('EmployeePicture')
Without result. Any idea?
Django's ORM is really powerful. You actually don't have to do joins like that, ever. When you access the field, the ORM performs the join on the fly, returning the result you want.
first_employee = Employee.objects.all().first()
employee_picture = first_employee.employee_img
And then employee_picture should have the picture of the first employee. It needed to internally do an inner join to figure that out, but Django hides all that for you. (I may have gotten your variable names wrong, sorry).
What select_related does is pre-fetch a relation for every object in a queryset. That means less trips back and forth to the database, which makes your functions faster. But, Django allows you to traverse relationships just by accessing attributes.
Try messing around with this stuff in your django shell. I would add django_extensions to your app (pip install django-extensions, then add django_extensions to your INSTALLED_APPS, then run python manage.py shell_plus). And if not, python manage.py shell works fine, but you have to import your models manually.
i am trying to insert some hex values into a field of a mysql db
this is the kind of value i need to insert :
['D\x93\xb4s\xa5\x9eM\\\x14\xf3*\x95\xf9\x83\x1d*%P\xdb\xa2', 'D\xbf\xef\xb0\xc8\xff\x17\xc6Y6\xc6\xb4,p\xaa\xb1\xf2V\xdaa', 'D\xd7~~\x02\xd3|}\xfcN\xc1\x03\x97\x07\xb5<U\x16Y\x9e', '\xf3\xb6\xc2,Y/[i\x98\x93\x9d\xb2R\x93\x84\x12W\x1a3\x19', '\xf3\xb7\xce\x1f-n\x89\xb6\x87K\x9dsf\xcb=w\xab\x1a\xa0<', '\xf3\xbf7\x04d\xe6\xdf\xf8"9\x1d\x05\x01\xe4\xd4\xb0\xad\x80\xc0\xf5']
this is my table
+--------------+--------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+-------+
| consensus | char(40) | NO | | NULL | |
| identityb32 | char(40) | NO | | NULL | |
| pubdate | char(40) | NO | | NULL | |
| dirport | char(6) | NO | | NULL | |
| ip | char(40) | NO | | NULL | |
| orport | char(40) | NO | | NULL | |
| identityhash | char(40) | NO | | NULL | |
| nick | char(40) | NO | | NULL | |
| version | char(40) | NO | | NULL | |
| flags | varchar(500) | NO | | NULL | |
| identity | char(40) | NO | | NULL | |
| digest | char(40) | NO | | NULL | |
| pubtime | char(40) | NO | | NULL | |
+--------------+--------------+------+-----+---------+-------+
13 rows in set (0.00 sec)
currently i am adding the hex data as i would do a normal string but this results in a non readable input being added:
D??s??M?*???*%P?
how can the hex data be added?
Check CHARSET
CREATE TABLE `t` (
`id` varchar(32) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
Maybe the way really is HEX() and UNHEX() functions. But, this post maybe help Inserting hex value mysql
Those aren't hex values.
I may be way off on this, but the only way I found to insert your values, was like this:
$string = 'D\x93\xb4s\xa5\x9eM\\\x14\xf3*\x95\xf9\x83\x1d*%P\xdb\xa2';
$pattern = '#\\\#';
$replacement = '\\\\\\';
$insert_value = preg_replace($pattern, $replacement, $string);
After which proceed to insert. You could use VARCHAR(256) for the column.
I'm positive there are a lot of better ways to go about this.
Apparently I'm on the python part of SO. I thought it was PHP. This is what I get for subscribing to multiple tags.
I have a ManyToMany field. Like this:
class Tag(models.Model):
books = models.ManyToManyField ('book.Book', related_name='vtags', through=TagBook)
class Book (models.Model):
nump = models.IntegerField (default=0, db_index=True)
I have around 450,000 books, and for some tags, it related around 60,000 books. When I did a query like:
tag.books.order_by('nump')[1:11]
It gets extremely slow, like 3-4 minutes.
But if I remove order_by, it run queries as normal.
The raw sql for the order_by version looks like this:
'SELECT `book_book`.`id`, ... `book_book`.`price`, `book_book`.`nump`,
FROM `book_book` INNER JOIN `book_tagbook` ON (`book_book`.`id` =
`book_tagbook`.`book_id`) WHERE `book_tagbook`.`tag_id` = 1 ORDER BY
`book_book`.`nump` ASC LIMIT 11 OFFSET 1'
Do you have any idea on this? How could I fix it? Thanks.
---EDIT---
Checked the previous raw query in mysql as #bouke suggested:
SELECT `book_book`.`id`, `book_book`.`title`, ... `book_book`.`nump`,
`book_book`.`raw_data` FROM `book_book` INNER JOIN `book_tagbook` ON
(`book_book`.`id` = `book_tagbook`.`book_id`) WHERE `book_tagbook`.`tag_id` = 1
ORDER BY `book_book`.`nump` ASC LIMIT 11 OFFSET 1;
11 rows in set (4 min 2.79 sec)
Then use explain to find out why:
+----+-------------+--------------+--------+---------------------------------------------+-----------------------+---------+-----------------------------+--------+---------------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+--------------+--------+---------------------------------------------+-----------------------+---------+-----------------------------+--------+---------------------------------+
| 1 | SIMPLE | book_tagbook | ref | book_tagbook_3747b463,book_tagbook_752eb95b | book_tagbook_3747b463 | 4 | const | 116394 | Using temporary; Using filesort |
| 1 | SIMPLE | book_book | eq_ref | PRIMARY | PRIMARY | 4 | legend.book_tagbook.book_id | 1 | |
+----+-------------+--------------+--------+---------------------------------------------+-----------------------+---------+-----------------------------+--------+---------------------------------+
2 rows in set (0.10 sec)
And for the table book_book:
mysql> explain book_book;
+----------------+----------------+------+-----+-----------+----------------+
| Field | Type | Null | Key | Default | Extra |
+----------------+----------------+------+-----+-----------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| title | varchar(200) | YES | | NULL | |
| href | varchar(200) | NO | UNI | NULL | |
..... skip some part.............
| nump | int(11) | NO | MUL | 0 | |
| raw_data | varchar(10000) | YES | | NULL | |
+----------------+----------------+------+-----+-----------+----------------+
24 rows in set (0.00 sec)
#!/usr/bin/python
import MySQLdb
import sys
import csv
csvfile='/home/arun/Documents/smartearn/Linux/SA/xbrlparser/UGT-2012-Concepts.csv'
def csvparsing():
spamReader = csv.reader(open(csvfile, 'rb'), delimiter=',', quotechar='"')
header=spamReader.next()
conn = MySQLdb.connect (host = "localhost",
user = "arunkamaly",
db = "se")
cursor=conn.cursor()
for clist in spamReader:
name=clist[1]+clist[2]
cursor.execute("""
INSERT INTO se_xbrl_concepts( xbrl_id,xbrl_name,xbrl_type,Xbrl_enumerations,Xbrl_substitutiongroup,Xbrl_balance, Xbrl_periodtype, Xbrl_IsAbstract,xbrl_label,Xbrl_desc)
values(clist[0],%s,%s,%s,%s,%s,%s,%s,clist[9],%s)""")(name,clist[3],clist[4],clist[5],clist[6],clist[7],clist[8],clist[10])
# print name
conn.commit()
cursor.close()
if(__name__=='__main__'):
csvparsing()
the mysql table is
Field | Type | Null | Key | Default | Extra |
+------------------------+--------------+------+-----+---------+----------------+
| xbrl_id | bigint(20) | NO | PRI | NULL | auto_increment |
| xbrl_name | varchar(255) | NO | | NULL | |
| xbrl_type | varchar(10) | NO | | NULL | |
| Xbrl_enumerations | varchar(25) | NO | | NULL | |
| Xbrl_substitutiongroup | varchar(255) | YES | | NULL | |
| Xbrl_balance | varchar(50) | YES | | NULL | |
| Xbrl_periodtype | varchar(25) | YES | | NULL | |
| Xbrl_IsAbstract | bit(1) | YES | | NULL | |
| xbrl_label | varchar(255) | YES | | NULL | |
| Xbrl_desc | varchar(45) | YES | | NULL | |
+------------------------+--------------+-----
Without the error trace, it's hard to say if this is the only error, but this is certainly one error:
cursor.execute("""
INSERT INTO se_xbrl_concepts( xbrl_id,xbrl_name,xbrl_type,Xbrl_enumerations,Xbrl_substitutiongroup,Xbrl_balance, Xbrl_periodtype, Xbrl_IsAbstract,xbrl_label,Xbrl_desc)
values(clist[0],%s,%s,%s,%s,%s,%s,%s,clist[9],%s)""")(name,clist[3],clist[4],clist[5],clist[6],clist[7],clist[8],clist[10])
You need a comma to separate your argument list from the string, and you need to move the parenthesis to the outside of that pair:
cursor.execute("""
INSERT INTO se_xbrl_concepts( xbrl_id,xbrl_name,xbrl_type,Xbrl_enumerations,Xbrl_substitutiongroup,Xbrl_balance, Xbrl_periodtype, Xbrl_IsAbstract,xbrl_label,Xbrl_desc)
values(clist[0],%s,%s,%s,%s,%s,%s,%s,clist[9],%s)""",(name,clist[3],clist[4],clist[5],clist[6],clist[7],clist[8],clist[10]))