We Keep Coding

PyQT Deleting QVBoxLayout In Order To Accomodate Updated Values from DB

So I have a problem with my PyQT5 interface is that when I load values from the database, it duplicates the in the view as it is being looped after button click.
Here is the code the populates a view when an item is inserted from the DB, it takes values from the DB. And displays it via loop.
def restart_program(self):
total, items = fetch_items()
for item in items:
item = str(item[0]) + ' - ' + str(item[2]) +'x'
self.b3 = QtWidgets.QPushButton(item)
self.v_box.addWidget(self.b3)
self.b3.clicked.connect(self.btn_click1)
curr_budget = fetch_budget()
curr_budget = curr_budget[0]
self.message2.setText("Total: " + str(total))
self.budget_status.setText("Budget: " + str(curr_budget))
self.message3.setText(" ")
The problem here is that.
Because of the view it doesnt delete the previous values. Resulting to something like this in the photo.
What I tried so far:
Getting the items and their frequencies and put them in a dictionary
Clearly didnt work as it just populated the db
had an idea to clear the view QVBoxLayout before so that when the view laods the data from db again, it wouldn't show the past inputs
But I'm not sure on how to implement #2. My full code can be seen here in the so_revision.py file

You could check how many elements you already have in your QVBoxLayout and remove them (just be careful not to remove your label etc) for eg:
def restart_program(self):
total, items = fetch_items()
for i in range(1, self.v_box.count()):
existing_item = self.v_box.itemAt(i).widget()
if existing_item:
existing_item.setParent(None)
for item in items:
item = str(item[0]) + ' - ' + str(item[2]) +'x'
self.b3 = QtWidgets.QPushButton(item)
self.v_box.addWidget(self.b3)
self.b3.clicked.connect(self.btn_click1)
curr_budget = fetch_budget()
curr_budget = curr_budget[0]
self.message2.setText("Total: " + str(total))
self.budget_status.setText("Budget: " + str(curr_budget))
self.message3.setText(" ")

How to write Xpath for dynamic value passing?

How to write Xpath for the following:
<input class="t2" style="background-color:#008000;" title="Jump to Detailed Analysis" type="button" value="Analyze" onclick="javascript:popAnalyze
("1622662"," SP0001622662","CS3_pro2_axeda6","5336293761");">
Highlighted values are there in some variable(st_name). Highlighted and Red colour rounded values will be changing dynamically.
I'm not able to get how to write Xpath for this.
import xlrd
path = r'C:\Users\tmou\PycharmProjects\Python\WebScraping\Book2.xlsx'
workbook = xlrd.open_workbook(path)
sheet = workbook.sheet_by_index(0)
for c in range(sheet.ncols):
for r in range(sheet.nrows):
st = (sheet.cell_value(r, c))
try:
if st == float(st):
st_string = int(st)
#variable = 1622662
#new_string = "javascript:popAnalyze("" + str(st_string) + "","SP0001622662","CS3_pro2_axeda6","5336293761");"
#driver.find_element_by_xpath("//input[#class='t2']/#onclick='" + st_string + "'").click()
#driver.find_element_by_xpath("//input[#value='Analyze' and contains(#onclick='" + st_string + "']")
#driver.find_element_by_xpath("//a[#title='" + st_string + "']")
HTML:
<input class="t2" style="background-color:#008000;" title="Jump to Detailed Analysis" type="button" value="Analyze" onclick="javascript:popAnalyze("1622662","SP0001622662","CS3_pro2_axeda6","5336293761");">

If the value that you are looking for is the one under onclick attribute then the following Xpath expression should work:
string(//input[#class='t2']/#onclick)
Edit 1
Can you try for XPath version 3 or lower:
//input[(#class='t2' and matches(#onclick,'1622662'))]
And for XPath version 3.1:
//input[#class='t2']/[matches(#onclick, '1622662')]

There are many ways to do this without the value of onclick, so don't bother even if it is dynamic, like shown below:
//input[#title='Jump to Detailed Analysis']
or
//input[#value='Analyze']
or
//input[#value='Analyze' and #title='Jump to Detailed Analysis']
EDIT 1:
You can use variable like shown below:
variable = "Analyze"
xpath = "//input[#value='" + variable + "']"
EDIT 2:
variable = 1622662
new_string = "javascript:popAnalyze("" + str(variable) + "","SP0001622662","CS3_pro2_axeda6","5336293761");"
EDIT 3:
variable = 1622662
xpath = "//input[#value='Analyze' and contains(#onclick,'" + str(variable) + "')]"
if driver.find_elements_by_xpath(xpath):
driver.find_element_by_xpath(xpath).click()
In the above code variable will be your dynamic value.
xpath variable will have a dynamic xpath based on the value of
variable
if driver.find_elements_by_xpath(xpath): will check if at least one
element with the xpath exit
if exists exits click on it

Use one of the following XPath :
//input[#value='Analyze' and contains(#onclick,'"+st_name+"')]
OR
//input[#title='Jump to Detailed Analysis' and contains(#onclick,'"+st_name+"')]
Final Code :
driver.find_element_by_xpath("//input[#title='Jump to Detailed Analysis' and contains(#onclick,'"+st_name+"')]")

Insert table data from website into table on my own website using Python and Beautiful Soup

I wrote some code that grabs the numbers I need from this website, but I don't know what to do next.
It grabs the numbers from the table at the bottom. The ones under calving ease, birth weight, weaning weight, yearling weight, milk and total maternal.
#!/usr/bin/python
import urllib2
from bs4 import BeautifulSoup
import pyperclip
def getPageData(url):
if not ('abri.une.edu.au' in url):
return -1
webpage = urllib2.urlopen(url).read()
soup = BeautifulSoup(webpage, "html.parser")
# This finds the epd tree and saves it as a searchable list
pedTreeTable = soup.find('table', {'class':'TablesEBVBox'})
# This puts all of the epds into a list.
# it looks for anything in pedTreeTable with an td tag.
pageData = pedTreeTable.findAll('td')
pageData.pop(7)
return pageData
def createPedigree(animalPageData):
''' make animalPageData much more useful. Strip the text out and put it in a dict.'''
animals = []
for animal in animalPageData:
animals.append(animal.text)
prettyPedigree = {
'calving_ease' : animals[18],
'birth_weight' : animals[19],
'wean_weight' : animals[20],
'year_weight' : animals[21],
'milk' : animals[22],
'total_mat' : animals[23]
}
for animalKey in prettyPedigree:
if animalKey != 'year_weight' and animalKey != 'dam':
prettyPedigree[animalKey] = stripRegNumber(prettyPedigree[animalKey])
return prettyPedigree
def stripRegNumber(animal):
'''returns the animal with its registration number stripped'''
lAnimal = animal.split()
strippedAnimal = ""
for word in lAnimal:
if not word.isdigit():
strippedAnimal += word + " "
return strippedAnimal
def prettify(pedigree):
''' Takes the pedigree and prints it out in a usable format '''
s = ''
pedString = ""
# this is also ugly, but it was the only way I found to format with a variable
cFormat = '{{:^{}}}'
rFormat = '{{:>{}}}'
#row 1 of string
s += rFormat.format(len(pedigree['calving_ease'])).format(
pedigree['calving_ease']) + '\n'
#row 2 of string
s += rFormat.format(len(pedigree['birth_weight'])).format(
pedigree['birth_weight']) + '\n'
#row 3 of string
s += rFormat.format(len(pedigree['wean_weight'])).format(
pedigree['wean_weight']) + '\n'
#row 4 of string
s += rFormat.format(len(pedigree['year_weight'])).format(
pedigree['year_weight']) + '\n'
#row 4 of string
s += rFormat.format(len(pedigree['milk'])).format(
pedigree['milk']) + '\n'
#row 5 of string
s += rFormat.format(len(pedigree['total_mat'])).format(
pedigree['total_mat']) + '\n'
return s
if __name__ == '__main__':
while True:
url = raw_input('Input a url you want to use to make life easier: \n')
pageData = getPageData(url)
s = prettify(createPedigree(pageData))
pyperclip.copy(s)
if len(s) > 0:
print 'the easy string has been copied to your clipboard'
I've just been using this code for easy copying and pasting. All I have to do is insert the URL, and it saves the numbers to my clipboard.
Now I want to use this code on my website; I want to be able to insert a URL in my HTML code, and it displays these numbers on my page in a table.
My questions are as follows:
How do I use the python code on the website?
How do I insert collected data into a table with HTML?

It sounds like you would want to use something like Django. Although the learning curve is a bit steep, it is worth it and it (of course) supports python.

Reduce RAM usage in Python script

I've written a quick little program to scrape book data off of a UNESCO website which contains information about book translations. The code is doing what I want it to, but by the time it's processed about 20 countries, it's using ~6GB of RAM. Since there are around 200 I need to process, this isn't going to work for me.
I'm not sure where all the RAM usage is coming from, so I'm not sure how to reduce it. I'm assuming that it's the dictionary that's holding all the book information, but I'm not positive. I'm not sure if I should simply make the program run once for each country, rather than processing the lot of them? Or if there's a better way to do it?
This is the first time I've written anything like this, and I'm a pretty novice, self-taught programmer, so please point out any significant flaws in the code, or improvement tips you have that may not directly relate to the question at hand.
This is my code, thanks in advance for any assistance.
from __future__ import print_function
import urllib2, os
from bs4 import BeautifulSoup, SoupStrainer
''' Set list of countries and their code for niceness in explaining what
is actually going on as the program runs. '''
countries = {"AFG":"Afghanistan","ALA":"Aland Islands","DZA":"Algeria"}
'''List of country codes since dictionaries aren't sorted in any
way, this makes processing easier to deal with if it fails at
some point, mid run.'''
country_code_list = ["AFG","ALA","DZA"]
base_url = "http://www.unesco.org/xtrans/bsresult.aspx?lg=0&c="
destination_directory = "/Users/robbie/Test/"
only_restable = SoupStrainer(class_="restable")
class Book(object):
def set_author(self,book):
'''Parse the webpage to find author names. Finds last name, then
first name of original author(s) and sets the Book object's
Author attribute to the resulting string.'''
authors = ""
author_last_names = book.find_all('span',class_="sn_auth_name")
author_first_names = book.find_all('span', attrs={\
'class':"sn_auth_first_name"})
if author_last_names == []: self.Author = [" "]
for author in author_last_names:
try:
first_name = author_first_names.pop()
authors = authors + author.getText() + ', ' + \
first_name.getText()
except IndexError:
authors = authors + (author.getText())
self.author = authors
def set_quality(self,book):
''' Check to see if book page is using Quality, then set it if
so.'''
quality = book.find_all('span', class_="sn_auth_quality")
if len(quality) == 0: self.quality = " "
else: self.quality = quality[0].contents[0]
def set_target_title(self,book):
target_title = book.find_all('span', class_="sn_target_title")
if len(target_title) == 0: self.target_title = " "
else: self.target_title = target_title[0].contents[0]
def set_target_language(self,book):
target_language = book.find_all('span', class_="sn_target_lang")
if len(target_language) == 0: self.target_language = " "
else: self.target_language = target_language[0].contents[0]
def set_translator_name(self,book) :
translators = ""
translator_last_names = book.find_all('span', class_="sn_transl_name")
translator_first_names = book.find_all('span', \
class_="sn_transl_first_name")
if translator_first_names == [] and translator_last_names == [] :
self.translators = " "
return None
for translator in translator_last_names:
try:
first_name = translator_first_names.pop()
translators = translators + \
(translator.getText() + ',' \
+ first_name.getText())
except IndexError:
translators = translators + \
(translator.getText())
self.translators = translators
def set_published_city(self,book) :
published_city = book.find_all('span', class_="place")
if len(published_city) == 0:
self.published_city = " "
else: self.published_city = published_city[0].contents[0]
def set_publisher(self,book) :
publisher = book.find_all('span', class_="place")
if len(publisher) == 0:
self.publisher = " "
else: self.publisher = publisher[0].contents[0]
def set_published_country(self,book) :
published_country = book.find_all('span', \
class_="sn_country")
if len(published_country) == 0:
self.published_country = " "
else: self.published_country = published_country[0].contents[0]
def set_year(self,book) :
year = book.find_all('span', class_="sn_year")
if len(year) == 0:
self.year = " "
else: self.year = year[0].contents[0]
def set_pages(self,book) :
pages = book.find_all('span', class_="sn_pagination")
if len(pages) == 0:
self.pages = " "
else: self.pages = pages[0].contents[0]
def set_edition(self, book) :
edition = book.find_all('span', class_="sn_editionstat")
if len(edition) == 0:
self.edition = " "
else: self.edition = edition[0].contents[0]
def set_original_title(self,book) :
original_title = book.find_all('span', class_="sn_orig_title")
if len(original_title) == 0:
self.original_title = " "
else: self.original_title = original_title[0].contents[0]
def set_original_language(self,book) :
languages = ''
original_languages = book.find_all('span', \
class_="sn_orig_lang")
for language in original_languages:
languages = languages + language.getText() + ', '
self.original_languages = languages
def export(self, country):
''' Function to allow us to easilly pull the text from the
contents of the Book object's attributes and write them to the
country in which the book was published's CSV file.'''
file_name = os.path.join(destination_directory + country + ".csv")
with open(file_name, "a") as by_country_csv:
print(self.author.encode('UTF-8') + " & " + \
self.quality.encode('UTF-8') + " & " + \
self.target_title.encode('UTF-8') + " & " + \
self.target_language.encode('UTF-8') + " & " + \
self.translators.encode('UTF-8') + " & " + \
self.published_city.encode('UTF-8') + " & " + \
self.publisher.encode('UTF-8') + " & " + \
self.published_country.encode('UTF-8') + " & " + \
self.year.encode('UTF-8') + " & " + \
self.pages.encode('UTF-8') + " & " + \
self.edition.encode('UTF-8') + " & " + \
self.original_title.encode('UTF-8') + " & " + \
self.original_languages.encode('UTF-8'), file=by_country_csv)
by_country_csv.close()
def __init__(self, book, country):
''' Initialize the Book object by feeding it the HTML for its
row'''
self.set_author(book)
self.set_quality(book)
self.set_target_title(book)
self.set_target_language(book)
self.set_translator_name(book)
self.set_published_city(book)
self.set_publisher(book)
self.set_published_country(book)
self.set_year(book)
self.set_pages(book)
self.set_edition(book)
self.set_original_title(book)
self.set_original_language(book)
def get_all_pages(country,base_url):
''' Create a list of URLs to be crawled by adding the ISO_3166-1_alpha-3
country code to the URL and then iterating through the results every 10
pages. Returns a string.'''
base_page = urllib2.urlopen(base_url+country)
page = BeautifulSoup(base_page, parse_only=only_restable)
result_number = page.find_all('td',class_="res1",limit=1)
if not result_number:
return 0
str_result_number = str(result_number[0].getText())
results_total = int(str_result_number.split('/')[1])
page.decompose()
return results_total
def build_list(country_code_list, countries):
''' Build the list of all the books, and return a list of Book objects
in case you want to do something with them in something else, ever.'''
for country in country_code_list:
print("Processing %s now..." % countries[country])
results_total = get_all_pages(country, base_url)
for url in range(results_total):
if url % 10 == 0 :
all_books = []
target_page = urllib2.urlopen(base_url + country \
+"&fr="+str(url))
page = BeautifulSoup(target_page, parse_only=only_restable)
books = page.find_all('td',class_="res2")
for book in books:
all_books.append(Book (book,country))
page.decompose()
for title in all_books:
title.export(country)
return
if __name__ == "__main__":
build_list(country_code_list,countries)
print("Completed.")

I guess I'll just list off some of the problems or possible improvements in no particular order:
Follow PEP 8.
Right now, you've got lots of variables and functions named using camel-case like setAuthor. That's not the conventional style for Python; Python would typically named that set_author (and published_country rather than PublishedCountry, etc.). You can even change the names of some of the things you're calling: for one, BeautifulSoup supports findAll for compatibility, but find_all is recommended.
Besides naming, PEP 8 also specifies a few other things; for example, you'd want to rewrite this:
if len(resultNumber) == 0 : return 0
as this:
if len(result_number) == 0:
return 0
or even taking into account the fact that empty lists are falsy:
if not result_number:
return 0
Pass a SoupStrainer to BeautifulSoup.
The information you're looking for is probably in only part of the document; you don't need to parse the whole thing into a tree. Pass a SoupStrainer as the parse_only argument to BeautifulSoup. This should reduce memory usage by discarding unnecessary parts early.
decompose the soup when you're done with it.
Python primarily uses reference counting, so removing all circular references (as decompose does) should let its primary mechanism for garbage collection, reference counting, free up a lot of memory. Python also has a semi-traditional garbage collector to deal with circular references, but reference counting is much faster.
Don't make Book.__init__ write things to disk.
In most cases, I wouldn't expect just creating an instance of a class to write something to disk. Remove the call to export; let the user call export if they want it to be put on the disk.
Stop holding on to so much data in memory.
You're accumulating all this data into a dictionary just to export it afterwards. The obvious thing to do to reduce memory is to dump it to disk as soon as possible. Your comment indicates that you're putting it in a dictionary to be flexible; but that doesn't mean you have to collect it all in a list: use a generator, yielding items as you scrape them. Then the user can iterate over it just like a list:
for book in scrape_books():
book.export()
…but with the advantage that at most one book will be kept in memory at a time.
Use the functions in os.path rather than munging paths yourself.
Your code right now is rather fragile when it comes to path names. If I accidentally removed the trailing slash from destinationDirectory, something unintended happens. Using os.path.join prevents that from happening and deals with cross-platform differences:
>>> os.path.join("/Users/robbie/Test/", "USA")
'/Users/robbie/Test/USA'
>>> os.path.join("/Users/robbie/Test", "USA") # still works!
'/Users/robbie/Test/USA'
>>> # or say we were on Windows:
>>> os.path.join(r"C:\Documents and Settings\robbie\Test", "USA")
'C:\\Documents and Settings\\robbie\\Test\\USA'
Abbreviate attrs={"class":...} to class_=....
BeautifulSoup 4.1.2 introduces searching with class_, which removes the need for the verbose attrs={"class":...}.
I imagine there are even more things you can change, but that's quite a few to start with.

What do you want the booklist for, in the end? You should export each book at the end of the "for url in range" block (inside it), and do without the allbooks dict. If you really need a list, define exactly what infos you will need, not keeping full Book objects.

Python ljust not displaying properly when including a lin

I am using Django creating a site for records for football teams, I have a "pretty" display with CSS, etc, but as a backup / old school version I am trying to have the code write the information to a basic .html file that is using rjust, ljust, etc to format text. In the code below if I remove the link code, and just display the string for the team's name everything lines up properly. Once I add the HTML for the link though the columns do not line up and are completely out of whack. What have I done wrong?
standings = Team.objects.filter(active=True).order_by('-wp')
output += '<pre>\n'
output += '%s %s %s %s\n' % (str('Rk').rjust(3), str('Team').ljust(50), str('W').rjust(2), str('L').rjust(2))
output += '%s %s %s %s\n' % (str('--').rjust(3), str('----').ljust(50), str('-').rjust(2), str('-').rjust(2))
for row in mpi:
the_team = "%s" % (row.slug, row.name)
output += '%s %s %s %s\n' % (str(row.rank).rjust(3), str(the_team).ljust(50), str(row.won).rjust(2), str(row.lost).rjust(2))
output += '</pre>'

The string "%s" contains some characters that aren't rendered on browser, you're formatting the source code, not the visualization.
Replace str(row.the_team).ljust(50) by str(row.the_team).ljust(50+len(row.slug)+15) because there are 15 invisible chars (ie. ) plus the slug.
Update: You may want to remove some str. If some value is already a string, you didn't need to (re)transform it in string again.. You may also split long lines in shorten ones.
output = '<pre>\n'
output += '%s %s%s%s\n' % ('Rk'.rjust(3), 'Team'.ljust(50), 'W'.rjust(2), 'L'.rjust(2))
output += '%s %s%s%s\n' % ('--'.rjust(3), '----'.ljust(50), '-'.rjust(2), '-'.rjust(2))
for team in teams:
link = '%s' % (team.slug, team.name)
link = link.ljust(50 + len(team.slug) + 15)
rank, won, lost = str(team.rank).rjust(3), str(team.won).rjust(2), str(team.lost).rjust(2)
output += '%s %s%s%s\n' % (rank, link, won, lost)
output += '</pre>'
print output