How i can i iterate through data on website - python

I want to extract users from a website bu using for loop, but i don't know how can i put correctly "i" instead of number 1
after i put
user_id = browser.find_element_by_xpath("(//div[#class='_gzjax'])["+str(i)+"]").text
Traceback (most recent call last): File "D:/Code/Python/Instagram
Unfollow/Instagram Unfollow.py", line 32, in
user_id = browser.find_element_by_xpath("(//div[#class='_gzjax'])["+str(i)+"]").text
File
"C:\Python33\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 293, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath) File "C:\Python33\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 752, in find_element
'value': value})['value'] File "C:\Python33\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 236, in execute
self.error_handler.check_response(response) File "C:\Python33\lib\site-packages\selenium\webdriver\remote\errorhandler.py",
line 192, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: Unable to
locate element:
{"method":"xpath","selector":"(//div[#class='gzjax'])[0]"}
Stacktrace:
at FirefoxDriver.prototype.findElementInternal (file:///c:/users/viktor/appdata/local/temp/tmp1r0vgw/extensions/fxdriver#googlecode.com/components/driver-component.js:10770)
at FirefoxDriver.prototype.findElement (file:///c:/users/viktor/appdata/local/temp/tmp1r0vgw/extensions/fxdriver#googlecode.com/components/driver-component.js:10779)
at DelayedCommand.prototype.executeInternal_/h (file:///c:/users/viktor/appdata/local/temp/tmp1r0vgw/extensions/fxdriver#googlecode.com/components/command-processor.js:12661)
at DelayedCommand.prototype.executeInternal_ (file:///c:/users/viktor/appdata/local/temp/tmp1r0vgw/extensions/fxdriver#googlecode.com/components/command-processor.js:12666)
at DelayedCommand.prototype.execute/< (file:///c:/users/viktor/appdata/local/temp/tmp1r0vgw/extensions/fxdriver#googlecode.com/components/command-processor.js:12608)

Just concatenate i (first cast it to string) to your desired string:
for i in range (0,100):
user_id = browser.find_element_by_xpath("(//div[#class='_gzjax'])["+str(i)+"]").text
print(user_id)

Try this. I think you concatenate variables into string with plus sign in python:
user_id = browser.find_element_by_xpath("(//div[#class='_gzjax'])["+i+"]").text
see how we add ("string" + variable + "string")

You would need to explicitly convert i from int to string. How about:
for i in range (0,100):
user_id = browser.find_element_by_xpath("(//div[#class='_gzjax'])["+str(i)+"]").text
print(user_id)

I don't know how many users there are on the page, but as it stands you're bound to 101.
Could just loop through the elements?
for i in browser.find_element_by_xpath("(//div[#class='_gzjax'])"):
user_id = i.text
print(user_id)
Something along those lines where you are iterating over items instead of indices.

Related

How can I load an image dynamically based on radiobutton selection

I am making adding a character selection to a queue accept tool using radiobuttons for the characters.
the character selection is based on object detection using cv2, and I need to make it so whatever character you select affects the file cv2 opens.
This is what I'm using:
def selection():
if select:
screenshot = wincap.get_screenshot()
# Import selection to Vision Class
test=0
test=v.get()
characterSearch = charSearch('./CharacterSelect/' + str(characters[test]+'.jpg'))
print('./CharacterSelect/' + str(characters[test]+'.jpg'))
characterMatch = characterSearch.find(screenshot, .63, 'rectangles')
if len(characterMatch):
charSelect_off()
window.after(500, selection)
#Stores interger values
v = IntVar()
v.set = ()
# Radio button loops for character select
for index in range(len(characters)):
Radiobutton(bottomframe,
compound = TOP,
image = portraits[index],
text = characters[index],
variable = v,
value = index,
).grid(
row = index//5,
column = index%5,
)
This errors saying it can't find the path.
the print however is able to print selection.
So I need to find a way to create a dynamic str with cv2.imread() in order to do this. or another option entirely, like if I can read it from a list of images loaded via tkinter, that would work too as I have a list built, so if i could index that, that would also be more optimal.
Edit:
Adding the error received:
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\lugex\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 1948, in __call__
return self.func(*args)
^^^^^^^^^^^^^^^^
File "C:\Users\lugex\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 861, in callit
func(*args)
File "c:\Users\lugex\OneDrive\Documents\Projects\Queue-Companion\main.py", line 182, in start
win32gui.SetForegroundWindow(handle)
pywintypes.error: (0, 'SetForegroundWindow', 'No error message is available')
[ WARN:0#11.286] global D:\a\opencv-python\opencv-python\opencv\modules\imgcodecs\src\loadsave.cpp (239) cv::findDecoder imread_('./CharacterSelect/Adriana.jpg'): can't open/read file: check file path/integrity
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users\lugex\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 1948, in __call__
return self.func(*args)
^^^^^^^^^^^^^^^^
File "C:\Users\lugex\AppData\Local\Programs\Python\Python311\Lib\tkinter\__init__.py", line 861, in callit
func(*args)
File "c:\Users\lugex\OneDrive\Documents\Projects\Queue-Companion\main.py", line 188, in start
search_on()
File "c:\Users\lugex\OneDrive\Documents\Projects\Queue-Companion\main.py", line 154, in search_on
Char_Search()
File "c:\Users\lugex\OneDrive\Documents\Projects\Queue-Companion\main.py", line 207, in Char_Search
search_off()
File "c:\Users\lugex\OneDrive\Documents\Projects\Queue-Companion\main.py", line 160, in search_off
charSelect_on()
File "c:\Users\lugex\OneDrive\Documents\Projects\Queue-Companion\main.py", line 165, in charSelect_on
selection()
File "c:\Users\lugex\OneDrive\Documents\Projects\Queue-Companion\main.py", line 220, in selection
characterSearch = charSearch('./CharacterSelect/' + str(characters[test]+'.jpg'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Users\lugex\OneDrive\Documents\Projects\Queue-Companion\vision.py", line 206, in __init__
self.accept_w = self.accept_img.shape[1]
^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'shape'
I was able to figure this out when I found this: https://stackoverflow.com/a/48562343/20593304
creating a dynamic path using the string I created.
charVariable=0
charVariable=v.get()
pathtoCharacters = os.path.join(os.getcwd(), "CharacterSearch", str(characters[charVariable]+".jpg"))
characterSearch = charSearch(pathtoCharacters)
which then implements to my class that performs the object detection.

can't enter text into element in selenium python

why i'm getting "InvalidArgumentException" when trying to enter text into angular input element.
NOTE: i'm getting xpath from csv file
class TimeSheet:
def __init__(self):
c_options = Options()
self.__driver = webdriver.Chrome(options=c_options)
self.__action = None
def enter_text_into_element(self):
text = self.__action['enter_text']
element = self.__driver.find_element_by_xpath(self.__action['xpath'])
element.clear()
print('text cleared')
element.send_keys(text)
def click_function(self):
click_status = False
element_clicked = self.__driver.find_element_by_xpath(self.__action['xpath'])
try:
element_clicked.click()
click_status = True
except Exception as ex:
print(ex)
return click_status
ts = TimeSheet()
for index, task in enumerate(tasks):
ts.set_action(task)
print(ts.get_step(), end=": ")
print(ts.get_action())
if ts.get_action() == 'click':
ts.click_function()
elif ts.get_action() == 'text':
ts.enter_text_into_element()
#Error Traceback
Traceback (most recent call last): File
"C:\PycharmProjects\testSelenium\timesheet_automation.py", line 116,
in
ts.enter_text_into_element() File "C:\PycharmProjects\testSelenium\timesheet_automation.py", line 82, in
enter_text_into_element
element = self.__driver.find_element_by_xpath(self.__action['xpath']) File
"C:\PycharmProjects\testSelenium\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath) File "C:\PycharmProjects\testSelenium\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 976, in find_element
return self.execute(Command.FIND_ELEMENT, { File "C:\PycharmProjects\testSelenium\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 321, in execute
self.error_handler.check_response(response) File "C:\PycharmProjects\testSelenium\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py",
line 242, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InvalidArgumentException: Message: invalid
argument: missing command parameters

Unable to get element text using Selenium with Chrome

I'm trying to scrape Merriam-Webster's Medical Dictionary for medical terms using Python and Chrome as the Selenium webdriver. So far, this is what I have:
from os import path
from selenium import webdriver
# Adding an ad-blocker to Chrome to speed up page load times
options = webdriver.ChromeOptions()
options.add_extension(path.abspath("ublock-origin.crx"))
# Declaring the Selenium webdriver
driver = webdriver.Chrome(chrome_options = options)
# Fetching the "A" terms as a test set
driver.get("https://www.merriam-webster.com/browse/medical/a")
scraped_words = [] # The list that will hold each word
page_num = 1
while page_num < 55: # There are 54 pages of "A" terms
try:
for i in range(4): # There are 3 columns per page of words
column = "/html/body/div/div/div[5]/div[2]/div[1]/div/div[3]/ul/li[" + str(i) + "]/a"
number_of_words = len(driver.find_elements_by_xpath(column))
for j in range(number_of_words):
word = driver.find_elements_by_xpath(column + "[" + str(j) + "]")
scraped_words.append(word)
driver.find_element_by_class_name("fa-angle-right").click() # Next page
page_num += 1 # Increment page number to keep track of current page
except:
driver.close()
# Write out words to a file
with open("medical_terms.dict", "w") as text_file:
for i in range(len(scraped_words)):
text_file.write(str(scraped_words[i]))
text_file.write("\n")
driver.close()
The above code fetches all the items, as the output of len(scraped_words) is the number expected. However, since I did not specify that I wanted to fetch the text of the elements, I get element identifiers (I think?) instead of text. If I decide to use word = driver.find_elements_by_xpath(column + "[" + str(j) + "]").text in order to specify that I want to get the text of the element, I get the following error:
Traceback (most recent call last):
File "mw_download.py", line 20, in <module>
number_of_words = len(driver.find_elements_by_xpath(column))
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 325, in find_elements_by_xpath
return self.find_elements(by=By.XPATH, value=xpath)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 817, in find_elements
'value': value})['value']
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: no such session
(Driver info: chromedriver=2.31.488774 (7e15618d1bf16df8bf0ecf2914ed1964a387ba0b),platform=Mac OS X 10.12.6 x86_64)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "mw_download.py", line 27, in <module>
driver.close()
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 541, in close
self.execute(Command.CLOSE)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: no such session
(Driver info: chromedriver=2.31.488774 (7e15618d1bf16df8bf0ecf2914ed1964a387ba0b),platform=Mac OS X 10.12.6 x86_64)
What is strange to me here is that the only code I change between runs is on line 22 yet the error message points out line 20 instead.
Any help in deciphering what's going on here and what I can do to fix it would be much appreciated! :+)
You just need to create a words list accessing your elements texts, changing:
word = driver.find_elements_by_xpath(column + "[" + str(j) + "]")
to:
word = [i.text for i in driver.find_elements_by_xpath(column + "[" + str(j) + "]")]
Because .find_elements_by_xpath will always return a list, accessing .text directly won't work.

Multiprocessing map unorderable?

I have an operation using a dictionnary that I want to parallelize, but multiprocessing.map is causing me headaches
def dict_segmentor(dictionnary,n_processes):
print("len dictionnary")
print(len(dictionnary))
if len(dictionnary) < n_processes:
seg_size = len(dictionnary)
else:
seg_size = len(dictionnary) // n_processes
print("segmenting dictionnary")
print("seg_size "+str(seg_size))
print("len(dictionnary) "+str(len(dictionnary)))
itemlist=list(dictionnary.items())
seg_ranges = [dict(itemlist[s:s+seg_size]) for s in range(1, len(dictionnary)+1, seg_size)]
print("finished")
return seg_ranges
def multiprocess_calc(n_processes, dictionnary,struc):
dictionnary=dictionnary
struc=struc
seg_ranges1 = dict_segmentor(dictionnary,n_processes)
#this is invoked to break the dict to be passed into dicts into a list. Works as expected
print("seg_range_check")
print("seg_ranges1 {}".format(type(seg_ranges1)))#Returns a dict as expected
print("seg_ranges1 {}".format(type(seg_ranges1[0])))#Returns a list as expected
print("seg_ranges1 {}".format(len(seg_ranges1))) #Returns expected len=1
print("seg_ranges1 {}".format(len(seg_ranges1[0]))) #Returns expected len
processes = multiprocessing.Pool(n_processes)
print("Mapping Building")
processes.map(Builder, seg_ranges1,1)
def main():
file_multiprocess = 'pref_multiprocess.csv'
n_CPUs = multiprocessing.cpu_count()
n_processes = n_CPUs-1
print("\nNumber of CPUs detected:", n_CPUs)
multiprocess_calc(n_processes, file_multiprocess,struc)
if __name__ == '__main__':
main()
Here is complete Traceback:
Traceback (most recent call last):
File "<ipython-input-37-d0279721826c>", line 1, in <module>
runfile('Pyscript.py', wdir='C:/Python Scripts')
File "C:\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 714, in runfile
execfile(filename, namespace)
File "C:\Anaconda3\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 89, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "Pyscript.py", line 1033, in <module>
main()
File "Pyscript.py", line 1025, in main
multiprocess_calc(n_processes, dictionnary,struc)
File "Pyscript.py", line 911, in multiprocess_calc
processes.map(Builder, seg_ranges1,1)
File "C:\Anaconda3\lib\multiprocessing\pool.py", line 260, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "C:\Anaconda3\lib\multiprocessing\pool.py", line 608, in get
raise self._value
TypeError: unorderable types: list() > int()
I don't understand, even by reading this carefully (https://docs.python.org/3.5/library/multiprocessing.html#module-multiprocessing).
Each chunk is a dict, so it should be sent through map to the builder.
But instead I get that stupid error and the traceback doesn't help. I looked into the code of pool.py but no luck,
and my builder is not involved since its first operation (a control printing) is not even displayed. The builder function seems to be totally ignored (and there is not even a syntax error)
So I conclude this is a problem with map.
In case I would have misunderstood the multiprocessing.map function and that it would first make a chunk, then iterate over it to apply map on each sub sub element, what function of multiprocessing could I use? Apply use only one thread. That would mean I should do this manually?
Please feel free to correct my code and give me some insights. Thanks by advance
Edit: here is the builder function:
def Builder(dictionary,struc=struc):
#def Builder(keys, dictionary=dictionary,struc=struc): #Alternative
#Note, I even tried to use only the keys, passing the dictionary from a global variable but it didn't work
print("Building Tree") #Not even displayed
print("type dictionary"+str(type(dictionary)))
frags=0
try:
if True:
print("Building")
#for id in keys: #Alternative
for id in dictionary:
seq=dictionary[id]
for i in range(3):
frags+=1
if len(seq)<=3:
break
seq=seq[i:-i]
struc.append(seq)
print("Number of frags found {}".format(frags))
except TypeError as e:
print (e)
print ("error in Builder")

WebDriver. Python. I want to store one of the option of drop-down menu by value as text, how to do it?

I want to store one of the option of drop-down menu by value as text. I select random option by next Python script:
#Random select option by value
assignJob = Select(driver.find_element_by_name('job[job_title]'))
jobValue = str(randint(1, 6))
assignJob.select_by_value(jobValue)
HTML code:
<select name="job[job_title]" class="formSelect valid" id="job_job_title">
<option value="" selected="selected">-- Select --</option>
<option value="1">Customer Service</option>
<option value="4">QA Engineer</option>
<option value="3">QA Manager</option>
<option value="2">SDET</option>
<option value="5">Software Developer</option>
<option value="6">Software Development Manager</option>
</select>
Error when I assign variable storedJob = driver.find_element_by_css_selector("#job_job_title option[value=jobValue]").text:
Traceback (most recent call last):
File "code.py", line 62, in <module>
storedJob = driver.find_element_by_css_selector("#job_job_title option[value
=jobValue]").text
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriv
er\remote\webdriver.py", line 365, in find_element_by_css_selector
return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriv
er\remote\webdriver.py", line 681, in find_element
{'using': by, 'value': value})['value']
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriv
er\remote\webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriv
er\remote\errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: u'no such element\n
(Session info: chrome=32.0.1700.102)\n (Driver info: chromedriver=2.8.241075,p
latform=Windows NT 6.1 SP1 x86)'
Code:
assignJob = Select(driver.find_element_by_name('job[job_title]'))
jobValue = str(randint(1, 6))
assignJob.select_by_value(jobValue)
storedJob = driver.find_element_by_css_selector("#job_job_title option[value=jobValue]").text
print storedJob
Another Error:
Traceback (most recent call last):
File "code.py", line 63, in <module>
storedJob = driver.find_element_by_css_selector(jobValueSelector).text
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriv
er\remote\webdriver.py", line 365, in find_element_by_css_selector
return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriv
er\remote\webdriver.py", line 681, in find_element
{'using': by, 'value': value})['value']
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriv
er\remote\webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriv
er\remote\errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidElementStateException: Message: u"invalid elem
ent state: Failed to execute query: '#job_job_title option[value=2]' is not a va
lid selector.\n (Session info: chrome=32.0.1700.102)\n (Driver info: chromedri
ver=2.8.241075,platform=Windows NT 6.1 SP1 x86)"
You could use a CSS selector, to locate that option and ask selenium to return its text.
driver.find_element_by_css_selector("#job_job_title option[value=2]").text
Replace 2 with the variable jobValue for you specific scenario.
EDIT
jobValue is a variable(str), you cannot directly pass a variable in Python.
jobValueSelector = "#job_job_title option[value='%s']" %jobValue
storedJob = driver.find_element_by_css_selector(jobValueSelector).text
print storedJob
The piece of code below is in Ruby.. I think Python and Ruby is almost the same.
select_list = driver.find_element(:id, 'job_job_title')
options = select_list.find_elements(:tag_name, 'option')
index = rand(1..options.count) # Get random number from index 1 to options.count
my_variable = options[index]
Let me try my skill in Python (not tested)! :)
select_list = driver.find_element_by_id("job_job_title")
options = select_list.find_elements_by_tag_name("option")
index = randint(1, len(options))
variable = options[index]

Categories