I am scraping a website and I used their url in order to paginate/get next values. I created an infinite loop so that it wont stop since the data in there are too many. But I dont know why my variable wont increment inside my loop. Also i want to break the infinite loop if there is no data rendered in the json results
Here is my code
pageNumber = 0
while True:
driver.implicitly_wait(3)
driver.get('https://reversewhois.domaintools.com/?ajax=mReverseWhois&call=ajaxUpdateRefinePreview&q=%5B%5B%5B%22whois%22%2C%222%22%2C%22VerifiedID%40SG-Mandatory%22%5D%5D%5D&to='+str(pageNumber))
time.sleep(3)
pre = driver.find_element_by_tag_name("pre").text
data = json.loads(pre)
table = data['results']
tables = pd.read_html(table,skiprows=1)
df = tables[-1]
# print(df.to_string(index=False))
pageNumber = pageNumber + 1 //it always prints 1
print(pageNumber)
continue // i want to loop again with the incremented pageNumber value
I think i did it wrong. In case, can you tell me what should i do? Thanks.
I don't see any problem with how you increment pageNumber.
In order to break the look, try:
table = data['results']
if not table:
break
Related
When I run this code to get the titles and links, I get 10X results. Any idea what I am doing wrong? Is there a way to stop the scraping when we reach the last result on the page?
Thanks!
while True:
web = 'https://news.google.com/search?q=weather&hl=en-US&gl=US&ceid=US%3Aen'
driver.get(web)
time.sleep(3)
titleContainers = driver.find_elements(by='xpath', value='//*[#class="DY5T1d RZIKme"]')
linkContainers = driver.find_elements(by='xpath', value='//*[#class="DY5T1d RZIKme"]')
if (len(titleContainers) != 0):
for i in range(len(titleContainers)):
counter = counter + 1
print("Counter: " + str(counter))
titles.append(titleContainers[i].text)
links.append(linkContainers[i].get_attribute("href"))
else:
break
You put yourself in an infinite loop, with that 'while True' statement. if (len(titleContainers) != 0): condition will always evaluate to True, once they're found in page (they're 100). You're not posting your full code as well, I imagine that counter, titles and links are lists defined somewhere in your code. You may want to test for counter to be less or equal to titleContainers length.
I have this code below that runs on a page, finds the element input-optionXXX where XXX is a 3 digit number that changes between 300 and 400, and clicks on it. I would like to store the numeric value that it finds on the page so that i can use that straight away in my other lines of code. Right now, in the section print(i), it shows the correct value. I just need some way of storing that value.
for i in range(300, 400):
try:
driver.find_element_by_id(f'input-option{i}').click()
print(i)
except NoSuchElementException:
continue
If I understand the intention of your code correctly, assigning the value to a new variable that is not i, and then breaking out of the for loop should do the trick. As long as you do not need to continue looking for "input-optionXXX", this should work, as break will only be reached if the try: succeeded which it would only do if it manages to find "input-optionXXX"
for i in range(300, 400):
try:
driver.find_element_by_id(f'input-option{i}').click()
number = i
break
except NoSuchElementException:
continue
The following code will gather data from an API and the try/except clause will help to handle several errors (from authentication, index, anything).
There's only one error (an authentication error) that I'm using the while True to repeat the API call to make sure I get the data and it will after a try or two. However if by any means I get another error, it'll be infinitely looping and I can't break it so it goes to the next iteration. I tried to create a counter and if the counter reaches to a number then (pass or continue or break) but it's not working.
## Create a array to loop to:
data_array_query = pd.date_range(start_date,end_date,freq='6H')
#This is my idea but is not working
#Create a counter
counter = 0
#Loop through the just created array
for idx in range(len(data_array_query)-1):
## If counter reaches move on to next for loop element
while True:
if counter>=5:
break
else:
try:
start_date = data_array_query[idx]
end_date = data_array_query[idx+1]
print('from',start_date,'to',end_date)
df = api.query(domain, site_slug, resolution, data_series_collection, start_date=str(start_date), end_date=str(end_date), env='prod', from_archive=True, phase='production').sort_index()
print(df.info())
break
except Exception as e:
print(e)
counter +=1
print(counter)
So the output of running this code for a couple of days show that when it runs 5 times (that's the counter max I set up) it does break but it breaks the whole loop and I only want it to move to the next date.
Any help will be appreciated,
You need to use a break statement to get out of a while True loop. pass and continue work for for loops that have a fixed number of iterations. While loops can go on forever (hence the different names)
I'm trying to fetch the web-table data using for loop. And the table has pagination up-to 42. here my code:
driver.get()
#identification and Locators
stack = driver.find_elements_by_xpath("//*[#id='container']/div/div[4]/table/tbody/tr/td[10]/div/ul/li")
quant = driver.find_elements_by_xpath("//*[#class='admin__data-grid-wrap']/table/tbody/tr/td[7]/div")
link = driver.find_elements_by_xpath("//*[#class='admin__data-grid-wrap']/table/tbody/tr/td[15]/a")
#Start a procedure
for i in driver.find_elements_by_xpath("//*[#id='container']/div/div[2]/div[2]/div[2]/div/div[2]/div/div[2]/button[2]"):
for steck,quanty,links in zip(stack,quant,link):
stuck = steck.text
quantity = quanty.text
linkes = links.get_attribute("href")
if stuck != 'No manage stock':
word = "Default Stock: "
stock = stuck.replace(word, '')
stocks = int(stock)
quanties = int(float(quantity))
if stocks < 0:
print(stocks,quanties,linkes)
stacks = abs(stocks)
total = stacks+quanties+1
print(total)
i.click()
driver.implicitly_wait(10)
print("Next Page")
This code fetches data from the 1st page. after clicking the next page. the 2nd for-loop didn't fetch 2nd-page data from web-table.
Most likely your query driver.find_elements_by_xpath("//*[#id='container']/div/div[2]/div[2]/div[2]/div/div[2]/div/div[2]/button[2]") only returns one element (the actual button to go to the next page) so I guess you should read the number of page and use it for an outer loop (or at least, you might have to rebind the selection on the HTML element representing the clickable button because it might change when a new page of the table is loaded) :
driver.get()
# Read the number of page and store it as an integer
nb_pages = int(driver.find_element_by_id('someId').text)
# Repeat your code (and rebind your selections, notably the one
# on the button to go to the next page) on each page of the table
for page in nb_pages:
# lines below are adapted from your code, I notably removed you first loop
stack = driver.find_elements_by_xpath("//*[#id='container']/div/div[4]/table/tbody/tr/td[10]/div/ul/li")
quant = driver.find_elements_by_xpath("//*[#class='admin__data-grid-wrap']/table/tbody/tr/td[7]/div")
link = driver.find_elements_by_xpath("//*[#class='admin__data-grid-wrap']/table/tbody/tr/td[15]/a")
# loop removed here (i also splited the string for readability
# (but it don't change the actual string value)
i = driver.find_elements_by_xpath(
"//*[#id='container']/div/div[2]/div[2]/div[2]"
"/div/div[2]/div/div[2]/button[2]")[0]
for steck, quanty, links in zip(stack, quant, link):
# your logic ...
# ...
# Load the next page:
i.click()
If you can't read the number of page, you may also use a while loop and exit it when you can't find a button to load the next page whit something like:
while True:
i = driver.find_elements_by_xpath(
"//*[#id='container']/div/div[2]/div[2]/div[2]"
"/div/div[2]/div/div[2]/button[2]")
if not i:
break
i = i[0]
# the rest of your logic
# ...
i.click()
This is only a guess (as we don't have a sample HTML code of the page / table structure that you are trying to use).
Hi everyone first time poster long term reader.
my problem is I want an else statment to loop inside a for loop.
I want the else statment to loop until the if statment above it is met?
can anyone tell me where I am going wrong I have tried so many diffrent ways including while loops inside if statments cant get my head round this ?
edit changed the code to a while loop not on prunes suggestion but cant escape the while loop
for url in results:
webdriver.get(url)
try:
liked = webdriver.find_elements_by_xpath("//span[#class=\"glyphsSpriteHeart__filled__24__red_5 u-__7\" and #aria-label=\"Unlike\"]")
maxcount = 0
while not liked:
sleep(1)
webdriver.find_element_by_xpath('//span/button/span').click()
numberoflikesgiven += 1
maxcount += 1
print'number of likes given : ',numberoflikesgiven
sleep(2)
webdriver.find_element_by_link_text('Next').click()
if maxcount >= 10:
print('max count reached .... moving on.')
continue
else:
print ('picture has already been liked...')
continue
Yes, you very clearly wrote an infinite loop:
while not liked:
... # no assignments to "liked"
You do not change the value of liked anywhere in the loop. You do not break the loop. Thus, you have an infinite loop. You need to re-evaluate liked on each iteration. A typical way to to this is to duplicate the evaluation at the bottom of the loop
sleep(2)
webdriver.find_element_by_link_text('Next').click()
liked = webdriver.find_elements_by_xpath(
"//span[#class=\"glyphsSpriteHeart__filled__24__red_5 u-__7\" \
and #aria-label=\"Unlike\"]")
Also note that your continue does nothing, as it's at the bottom of the loop. Just let your code reach the bottom naturally, and the whilewill continue on its own. If you intended to go to the next iteration offor url in results, then you need tobreakthewhile` loop instead.