We Keep Coding

How to get a reference to individual buttons created in a loop in tkinter?

Attempting to create a book issue page.
I get the data (the document variable) displayed as an array of n-ples, and add the borrow/return buttons in the loop with these data fields. However, since I create multiple instances of the same button variable as seen below, I have no way of knowing which one is clicked. Here is the code:
document = self.app.Member(MemberVerification.mem_id, self.app).searchDocument(SearchBooks.inputvalues, "Books")
logger.info("Results: " + str(document) + " for string filters: " + str(filters))
cells={}
for i in range(len(document)): # Rows
self.borrow = tk.Button(self, text="Borrow", command= lambda: self.checkBorrows(document[i], self.controller, i))
self.retbook = tk.Button(self, text="Return")
for j in range(1, len(document[1])): # Columns
try:
b = tk.Entry(self,justify=tk.CENTER)
b.grid(row=i, column=j)
b.insert(tk.END, str(document[i][j]))
cells[(i, j)] = b
except Exception as e:
logger.error("Error in populateTable: " + str(e) + traceback.format_exc())
sys.exit(-1)
if i!=0:
self.borrow.grid(row=i, column=j+1)
self.borrowbuttons.append(self.borrow)
self.retbook.grid(row=i, column=j+2)
A little explanation - i == 0 is where the column names come in. The button in question is self.borrow. I tried maintaining a list of them to search which button it is, but that didn't work. I would like to be able to maintain a reference to the specific button out of the 3 that have been created, so that the callback function checkBorrows can know which book has to be borrowed. The i parameter was just experimentation, doesn't seem to pass the index in due to the async nature of the button click listener
Any suggestions? I tried maintaining a class variable, but that always ends up referring to the last button, since the callback is technically async.
Thanks!

How to call a view from within itself in a Django REST API?

I have this function in my Django REST API that handles the insertion of products. Sometimes a product will be a variational product (when product.type = 3), in which case I get all the permutations of these variations and want to insert a new product corresponding to each permutation into the database.
#api_view(['POST'])
def upsertProduct(request):
productData = request.data['product']
variations = productData.pop('variations', [])
variationTypes = productData.pop('variationTypes', [])
product, created = Product.objects.update_or_create(pk=productData['id'], defaults=productData)
if (productData['type'].id == 3):
variations_by_type = []
for variationType in variationTypes:
variations_by_type.append([variation for variation in variations if variation['variationType'] == variationType['id']])
combinations = list(itertools.product(*variations_by_type))
for combination in combinations:
productData['name'] = product.name + ' (' + ' | '.join(' : '.join(i['name'] for i in item) for item in combination) + ')'
productData['type'] = {'id': 5}
productData['parent'] = product.id
#Recursive call should go here
#upsertProduct(request={'data': {'product': productData}})
My first attempt was to simply call the function as I did in the commented line. Django returned the following
AssertionError: The request argument must be an instance of
django.http.HttpRequest, not builtins.dict.
I then attempted making use of this HttpRequest object from Django, but can not figure out how to use it properly for this use case. I've also tried the standard request library in Python, without success, and I am also pretty sure that wouldn't be an ideal solution even if it did work.

In general, I wouldn't create directly a HttpRequest object that would be used by a view somewhere else, because it contains much more information than the payload data, such as session, user, middleware, context info, etc.
What you could do in your code, is keeping the single request variable as is and make another function that handles a dictionary of data.
For example, taking what you wrote, that would give something like:
def handle_data(productData):
"""
Handle a dictionary of data
"""
product, created = Product.objects.update_or_create(pk=productData['id'], defaults=productData)
return product
#api_view(['POST'])
def upsertProduct(request):
productData = request.data['product']
variations = productData.pop('variations', [])
variationTypes = productData.pop('variationTypes', [])
product = handle_data(productData)
if (productData['type'].id == 3):
variations_by_type = []
for variationType in variationTypes:
variations_by_type.append([variation for variation in variations if variation['variationType'] == variationType['id']])
combinations = list(itertools.product(*variations_by_type))
for combination in combinations:
productData['name'] = product.name + ' (' + ' | '.join(' : '.join(i['name'] for i in item) for item in combination) + ')'
productData['type'] = {'id': 5}
productData['parent'] = product.id
handle_data(productData)

Why does Python(3.7) keep overriding key:value pairs in my nested dictionary?

I want to preface this by saying I replicated the spirit of the code in the python terminal by writing:
A = {}
A["Q"] = {}
A["Q"]["creation"] = {}
A["Q"]["creation"]["reactants_1"] = ["R1","R2"]
printing A gives what one would expect:
{"Q" : {"creation" : {"reactants_1" : ["R1","R2"]}}}
then adding:
A["Q"]["creation"]["reactants_2"] = ["R3","R4"]
yields:
{"Q" : {"creation" : {"reactants_1" : ["R1","R2"], "reactants_2" : ["R3","R4"]}}}
However I have a function displayed below which, when run correctly assigns the first few key:value pairs but when it loops over to the second value of x it replaces the first key:value pair it wrote earlier with a new one despite the key being different.
using the above example we would only get:
{"Q" : {"creation" : {"reactants_2" : ["R3","R4"]}}}
Reactions is an array containing things like "e+e+p=e+H_1" in the format:
["e+e+p=e+H_1", "e+e+p=e+H_2",...,"e+H_10=e+e+p"]
Species is an array like:
[["e", 100000],["p", 100000],...]
where for now we only care about the string part.
diff_input is an empty dictionary to begin with
reactions_const contains, for each reaction, the left and right sides separate - seen as [x][0][0] and [x][0][1] early in the function as well as some other information that is not important for now.
rates_names is an array of unique identifiers for each reaction i can use later hence using the dictionary approach.
the print statements are all me trying to figure out why it isn't working
def rate_eqns(diff_input, reactions, species, reactions_const,
rates_names):
for x in range(len(reactions)):
# for each reaction
# split the left and right hand side up into lists of their
# respective species involved for counting later
print("reaction: " + str(x) + " " + str(reactions[x]))
species_lhs = reactions_const[x][0][0].split('+')
print("LHS = " + str(species_lhs))
species_rhs = reactions_const[x][0][1].split('+')
for y in range(len(species)):
# For each species, create a sub-dictionary
diff_input[species[y][0]] = {}
# create sub-dictionaries in each species for creation, destruction and preservation/neutral paths
diff_input[species[y][0]]["destruction"] = {}
diff_input[species[y][0]]["creation"] = {}
diff_input[species[y][0]]["balanced"] = {}
# check if species occurs in each reaction
if species[y][0] in reactions[x][0]:
# if you start with more of it than you finish its destruction
if species_lhs.count(species[y][0]) > species_rhs.count(species[y][0]):
# if true: add an entry to the dictionary which holds the reaction identifier
# bound to the destruction/creation/balanced identifier bound to the species identifier.
print("species:" + str(species[y][0]) + " found net loss from reactants")
print("LHS = " + str(species_lhs))
print("RHS = " + str(species_rhs))
print("reaction designation = " + str(rates_names[x]) + " Destruction of species")
print("-------------")
diff_input[species[y][0]]["destruction"][rates_names[x]] = species_lhs
elif species_lhs.count(species[y][0]) == species_rhs.count(species[y][0]):
print("species:" + str(species[y][0]) + " found no change in number")
print("LHS = " + str(species_lhs))
print("RHS = " + str(species_rhs))
print("reaction designation = " + str(rates_names[x]) + " preservation of species")
diff_input[species[y][0]]["balanced"][rates_names[x]] = species_lhs
elif species_lhs.count(species[y][0]) < species_rhs.count(species[y][0]):
print("species:" + str(species[y][0]) + " found net gain from reactants")
print("LHS = " + str(species_lhs))
print("RHS = " + str(species_rhs))
print("reaction designation = " + str(rates_names[x]) + " creation of species")
diff_input[species[y][0]]["creation"][rates_names[x]] = species_lhs
# else:
# print(str(species[y][0]) + " not found in reaction")
print(diff_input)
a = input("press return to continue")
with open('diff_input.txt', 'w') as file:
file.write(str(diff_input))
return diff_input
the file saving part is optional, has anyone else every encountered a dictionary overriding existing keys with new keys?
Thanks for your patience and I appreciate any advice on my formatting (I tried to make it as good as possible without including the rest of the script)

The actual value of your species[1][0] must happen to be equal to species[0][0], so that when it loops over to the second value of x, the assignment diff_input[species[y][0]] = {} would overwrite the sub-dict of the previous iteration since species[y][0] stays the same.
To put it more simply using your example code, in your outer loop you are making the following initialization:
A["Q"] = {}
A["Q"]["creation"] = {}
so even if your inner loop assigns some values to the sub-dict:
A["Q"]["creation"]["reactants_1"] = ["R1","R2"]
A["Q"] = {} would overwrite it in the next iteration as long as "Q" continues to be the main key for the assignment.

Creating new table while iterating through a queryset in django

This is a newbie question, but despite reading https://docs.djangoproject.com/en/dev/ref/models/instances/#saving-objects , I'm not quite sure how to do this. I have an existing table where I would like to iterate through all its records, and save certain info to a second table. I have the following model:
class myEmails(models.Model):
text = models.CharField(max_length=1200)
In my view I have:
def getMyMessages(request):
from django_mailbox.models import Message
from get_new_emails.models import myEmails
import re
qs = Message.objects.all()
count = 0
output = ""
for i in qs:
count += 1
output = output + str(count) + " TEXT: " + i.text + '<br>' + '<br>'
return HttpResponse(output)
How can I modify my view to save "i.text" to the text field of the 'myEmails' table

You can create new objects and save them to the database afterwards using save():
for i in qs:
obj = myEmails(text=i.text)
obj.save()

Reduce RAM usage in Python script

I've written a quick little program to scrape book data off of a UNESCO website which contains information about book translations. The code is doing what I want it to, but by the time it's processed about 20 countries, it's using ~6GB of RAM. Since there are around 200 I need to process, this isn't going to work for me.
I'm not sure where all the RAM usage is coming from, so I'm not sure how to reduce it. I'm assuming that it's the dictionary that's holding all the book information, but I'm not positive. I'm not sure if I should simply make the program run once for each country, rather than processing the lot of them? Or if there's a better way to do it?
This is the first time I've written anything like this, and I'm a pretty novice, self-taught programmer, so please point out any significant flaws in the code, or improvement tips you have that may not directly relate to the question at hand.
This is my code, thanks in advance for any assistance.
from __future__ import print_function
import urllib2, os
from bs4 import BeautifulSoup, SoupStrainer
''' Set list of countries and their code for niceness in explaining what
is actually going on as the program runs. '''
countries = {"AFG":"Afghanistan","ALA":"Aland Islands","DZA":"Algeria"}
'''List of country codes since dictionaries aren't sorted in any
way, this makes processing easier to deal with if it fails at
some point, mid run.'''
country_code_list = ["AFG","ALA","DZA"]
base_url = "http://www.unesco.org/xtrans/bsresult.aspx?lg=0&c="
destination_directory = "/Users/robbie/Test/"
only_restable = SoupStrainer(class_="restable")
class Book(object):
def set_author(self,book):
'''Parse the webpage to find author names. Finds last name, then
first name of original author(s) and sets the Book object's
Author attribute to the resulting string.'''
authors = ""
author_last_names = book.find_all('span',class_="sn_auth_name")
author_first_names = book.find_all('span', attrs={\
'class':"sn_auth_first_name"})
if author_last_names == []: self.Author = [" "]
for author in author_last_names:
try:
first_name = author_first_names.pop()
authors = authors + author.getText() + ', ' + \
first_name.getText()
except IndexError:
authors = authors + (author.getText())
self.author = authors
def set_quality(self,book):
''' Check to see if book page is using Quality, then set it if
so.'''
quality = book.find_all('span', class_="sn_auth_quality")
if len(quality) == 0: self.quality = " "
else: self.quality = quality[0].contents[0]
def set_target_title(self,book):
target_title = book.find_all('span', class_="sn_target_title")
if len(target_title) == 0: self.target_title = " "
else: self.target_title = target_title[0].contents[0]
def set_target_language(self,book):
target_language = book.find_all('span', class_="sn_target_lang")
if len(target_language) == 0: self.target_language = " "
else: self.target_language = target_language[0].contents[0]
def set_translator_name(self,book) :
translators = ""
translator_last_names = book.find_all('span', class_="sn_transl_name")
translator_first_names = book.find_all('span', \
class_="sn_transl_first_name")
if translator_first_names == [] and translator_last_names == [] :
self.translators = " "
return None
for translator in translator_last_names:
try:
first_name = translator_first_names.pop()
translators = translators + \
(translator.getText() + ',' \
+ first_name.getText())
except IndexError:
translators = translators + \
(translator.getText())
self.translators = translators
def set_published_city(self,book) :
published_city = book.find_all('span', class_="place")
if len(published_city) == 0:
self.published_city = " "
else: self.published_city = published_city[0].contents[0]
def set_publisher(self,book) :
publisher = book.find_all('span', class_="place")
if len(publisher) == 0:
self.publisher = " "
else: self.publisher = publisher[0].contents[0]
def set_published_country(self,book) :
published_country = book.find_all('span', \
class_="sn_country")
if len(published_country) == 0:
self.published_country = " "
else: self.published_country = published_country[0].contents[0]
def set_year(self,book) :
year = book.find_all('span', class_="sn_year")
if len(year) == 0:
self.year = " "
else: self.year = year[0].contents[0]
def set_pages(self,book) :
pages = book.find_all('span', class_="sn_pagination")
if len(pages) == 0:
self.pages = " "
else: self.pages = pages[0].contents[0]
def set_edition(self, book) :
edition = book.find_all('span', class_="sn_editionstat")
if len(edition) == 0:
self.edition = " "
else: self.edition = edition[0].contents[0]
def set_original_title(self,book) :
original_title = book.find_all('span', class_="sn_orig_title")
if len(original_title) == 0:
self.original_title = " "
else: self.original_title = original_title[0].contents[0]
def set_original_language(self,book) :
languages = ''
original_languages = book.find_all('span', \
class_="sn_orig_lang")
for language in original_languages:
languages = languages + language.getText() + ', '
self.original_languages = languages
def export(self, country):
''' Function to allow us to easilly pull the text from the
contents of the Book object's attributes and write them to the
country in which the book was published's CSV file.'''
file_name = os.path.join(destination_directory + country + ".csv")
with open(file_name, "a") as by_country_csv:
print(self.author.encode('UTF-8') + " & " + \
self.quality.encode('UTF-8') + " & " + \
self.target_title.encode('UTF-8') + " & " + \
self.target_language.encode('UTF-8') + " & " + \
self.translators.encode('UTF-8') + " & " + \
self.published_city.encode('UTF-8') + " & " + \
self.publisher.encode('UTF-8') + " & " + \
self.published_country.encode('UTF-8') + " & " + \
self.year.encode('UTF-8') + " & " + \
self.pages.encode('UTF-8') + " & " + \
self.edition.encode('UTF-8') + " & " + \
self.original_title.encode('UTF-8') + " & " + \
self.original_languages.encode('UTF-8'), file=by_country_csv)
by_country_csv.close()
def __init__(self, book, country):
''' Initialize the Book object by feeding it the HTML for its
row'''
self.set_author(book)
self.set_quality(book)
self.set_target_title(book)
self.set_target_language(book)
self.set_translator_name(book)
self.set_published_city(book)
self.set_publisher(book)
self.set_published_country(book)
self.set_year(book)
self.set_pages(book)
self.set_edition(book)
self.set_original_title(book)
self.set_original_language(book)
def get_all_pages(country,base_url):
''' Create a list of URLs to be crawled by adding the ISO_3166-1_alpha-3
country code to the URL and then iterating through the results every 10
pages. Returns a string.'''
base_page = urllib2.urlopen(base_url+country)
page = BeautifulSoup(base_page, parse_only=only_restable)
result_number = page.find_all('td',class_="res1",limit=1)
if not result_number:
return 0
str_result_number = str(result_number[0].getText())
results_total = int(str_result_number.split('/')[1])
page.decompose()
return results_total
def build_list(country_code_list, countries):
''' Build the list of all the books, and return a list of Book objects
in case you want to do something with them in something else, ever.'''
for country in country_code_list:
print("Processing %s now..." % countries[country])
results_total = get_all_pages(country, base_url)
for url in range(results_total):
if url % 10 == 0 :
all_books = []
target_page = urllib2.urlopen(base_url + country \
+"&fr="+str(url))
page = BeautifulSoup(target_page, parse_only=only_restable)
books = page.find_all('td',class_="res2")
for book in books:
all_books.append(Book (book,country))
page.decompose()
for title in all_books:
title.export(country)
return
if __name__ == "__main__":
build_list(country_code_list,countries)
print("Completed.")

I guess I'll just list off some of the problems or possible improvements in no particular order:
Follow PEP 8.
Right now, you've got lots of variables and functions named using camel-case like setAuthor. That's not the conventional style for Python; Python would typically named that set_author (and published_country rather than PublishedCountry, etc.). You can even change the names of some of the things you're calling: for one, BeautifulSoup supports findAll for compatibility, but find_all is recommended.
Besides naming, PEP 8 also specifies a few other things; for example, you'd want to rewrite this:
if len(resultNumber) == 0 : return 0
as this:
if len(result_number) == 0:
return 0
or even taking into account the fact that empty lists are falsy:
if not result_number:
return 0
Pass a SoupStrainer to BeautifulSoup.
The information you're looking for is probably in only part of the document; you don't need to parse the whole thing into a tree. Pass a SoupStrainer as the parse_only argument to BeautifulSoup. This should reduce memory usage by discarding unnecessary parts early.
decompose the soup when you're done with it.
Python primarily uses reference counting, so removing all circular references (as decompose does) should let its primary mechanism for garbage collection, reference counting, free up a lot of memory. Python also has a semi-traditional garbage collector to deal with circular references, but reference counting is much faster.
Don't make Book.__init__ write things to disk.
In most cases, I wouldn't expect just creating an instance of a class to write something to disk. Remove the call to export; let the user call export if they want it to be put on the disk.
Stop holding on to so much data in memory.
You're accumulating all this data into a dictionary just to export it afterwards. The obvious thing to do to reduce memory is to dump it to disk as soon as possible. Your comment indicates that you're putting it in a dictionary to be flexible; but that doesn't mean you have to collect it all in a list: use a generator, yielding items as you scrape them. Then the user can iterate over it just like a list:
for book in scrape_books():
book.export()
…but with the advantage that at most one book will be kept in memory at a time.
Use the functions in os.path rather than munging paths yourself.
Your code right now is rather fragile when it comes to path names. If I accidentally removed the trailing slash from destinationDirectory, something unintended happens. Using os.path.join prevents that from happening and deals with cross-platform differences:
>>> os.path.join("/Users/robbie/Test/", "USA")
'/Users/robbie/Test/USA'
>>> os.path.join("/Users/robbie/Test", "USA") # still works!
'/Users/robbie/Test/USA'
>>> # or say we were on Windows:
>>> os.path.join(r"C:\Documents and Settings\robbie\Test", "USA")
'C:\\Documents and Settings\\robbie\\Test\\USA'
Abbreviate attrs={"class":...} to class_=....
BeautifulSoup 4.1.2 introduces searching with class_, which removes the need for the verbose attrs={"class":...}.
I imagine there are even more things you can change, but that's quite a few to start with.

What do you want the booklist for, in the end? You should export each book at the end of the "for url in range" block (inside it), and do without the allbooks dict. If you really need a list, define exactly what infos you will need, not keeping full Book objects.