We Keep Coding

Exception doesn't print out the desired output and quits the program instead in Selenium

I am trying to scrape certain information from a webpage [https://reality.idnes.cz/rk/detail/m-m-reality-holding-a-s/5a85b582a26e3a321d4f2700/]. I have a list of links from which I need to go through all of them. Each links contains the same information about a different company. However few of the companies didn't add phone number for example. However if that happens the whole program is terminated with an exception error. This is my code :
for link in link_list:
try:
driver.get(', '.join(link))
time.sleep(2)
information_list = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "h1.b-annot__title.mb-5")))
title = driver.find_element_by_css_selector("h1.b-annot__title.mb-5")
information_list = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "span.btn__text")))
offers = driver.find_element_by_css_selector("span.btn__text")
information_list = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "p.font-sm")))
addresses = driver.find_element_by_css_selector("p.font-sm")
information_list = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "a.item-icon.measuring-data-layer")))
phone_number = driver.find_element_by_css_selector("a.item-icon.measuring-data-layer")
information_list = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "a.item-icon")))
email = driver.find_element_by_css_selector("a.item-icon")
print(title.text, " ", offers.text, " ", addresses.text, " ", phone_number.text, " ", email.text)
writer.writerow([title.text, offers.text, addresses.text, phone_number.text, email.text])
except Exception:
print(title.text, " ", offers.text, " ", addresses.text, " ", phone_number.text, " ", email.text)
writer.writerow([title.text, offers.text, addresses.text, phone_number.text, email.text])
continue
As you can see I am trying to prevent the whole program from terminating. I have tried adding the Exception > continue so the program wouldn't get terminated but I noticed that even though the program doesn't terminate none of the information is then scraped from the webpages that the exception occurred on. I am trying to prevent the loss of the data by requesting the desired output once again when the exception occurs so it would get printed out and saved into a CSV file with the missing information as an empty slot.
However the whole problem is that when I request the output in the exception the exception once again terminates the whole program instead of printing out what it knows and moving on with "continue". Now my question is why is that happening? Why doesn't the program print out the output and then won't follow the "continue" and terminates itself instead? How can one print the output that the program got without the missing information and prevent the program from terminating?

when it hits the exception the continue will allow subsequent lines to execute, it will NOT pick up where it left off.
See This answer: https://stackoverflow.com/a/19523054/1387701
No, you cannot do that. That's just the way Python has its syntax. Once you exit a try-block because of an exception, there is no way back in.
So I think this might help the problem you're finding, as an example, when phone number is missing/not found:
for link in link_list:
driver.get(', '.join(link))
time.sleep(2)
information_list = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "h1.b-annot__title.mb-5")))
title = driver.find_element_by_css_selector("h1.b-annot__title.mb-5")
information_list = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "span.btn__text")))
offers = driver.find_element_by_css_selector("span.btn__text")
information_list = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "p.font-sm")))
addresses = driver.find_element_by_css_selector("p.font-sm")
try:
information_list = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "a.item-icon.measuring-data-layer")))
phone_number = driver.find_element_by_css_selector("a.item-icon.measuring-data-layer")
except Exception as e:
print ("Phone number exception: %s" % str(e) )
continue
information_list = wait.until(ec.presence_of_element_located((By.CSS_SELECTOR, "a.item-icon")))
email = driver.find_element_by_css_selector("a.item-icon")
print(title.text, " ", offers.text, " ", addresses.text, " ", phone_number.text, " ", email.text)
writer.writerow([title.text, offers.text, addresses.text, phone_number.text, email.text])
You could looks for the specifc exception (TimeoutException from the wait, or NoSuchElementException on the find element command) to be a bit more specific for each element.
Or you could use an if find_elements_by_css_Selector size is > 0 before proceeding, but i'd prefer the try catch myself.

Make tweepy search for the newest mentions instead of the oldest

Today I wrote a twitter bot that replies anybody who mentions it with a random image from a folder.
The problem here is that I'm a newbie in python and I don't know how to make it funcitonal at all. When I started running it, the bot started replying all the mentions from other users (I'm using an old account a friend gave to me), and that's not precisely what I want, even if it's working, but not as I desire.
The bot is replying all the mentions from the very beggining and it won't stop until all these are replied (the bot is now turned off, I don't want to annoy anybody)
How can I make it to only reply to latest mentions instead of the first ones?
here's the code:
import tweepy
import logging
from config import create_api
import time
import os
import random
from datetime import datetime
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()
api = create_api()
imagePath = random.choice(os.listdir("images/"))
while True:
for tweet in tweepy.Cursor(api.mentions_timeline).items():
try:
imagePath = random.choice(os.listdir("images/"))
tweetId = tweet.user.id
username = tweet.user.screen_name
api.update_with_media('images/' + imagePath, "#" + username + " ", in_reply_to_status_id=tweet.id)
print('Replying to ' + username + 'with ' + imagePath)
except tweepy.TweepError as e:
print(e.reason)
except StopIteration:
break
time.sleep(12)
Thanks in advance.

I don't have the ability to test this code currently but this should work.
Instead of iterating over every tweet, it turns the iterator that tweepy.Cursor returns into a list and then just gets the last item in that list.
api = create_api()
imagePath = random.choice(os.listdir("images/"))
while True:
tweet_iterator = tweepy.Cursor(api.mentions_timeline).items()
latest_tweet = list(tweet_iterator)[-1]
try:
imagePath = random.choice(os.listdir("images/"))
tweetId = latest_tweet.user.id
username = latest_tweet.user.screen_name
api.update_with_media('images/' + imagePath, "#" + username + " ", in_reply_to_status_id=latest_tweet.id)
print('Replying to ' + username + 'with ' + imagePath)
except tweepy.TweepError as e:
print(e.reason)
except StopIteration:
break
time.sleep(12)
You will also want to keep track of what user you last replied to, so you don't just keep spamming the same person over and over.
This isn't the most efficient way of doing it but should be easy enough to understand:
latest_user_id = None
while True:
# Rest of the code
try:
if latest_user_id == latest_tweet.user.id:
# don't do anything
else:
latest_user_id = latest_tweet.user.id
# the rest of your code

Starting threads in a for cycle produces multiple results

-I found the problem!- In function SendMessage I was using UserID (with capital letters) instead of userid (which was the actual parameter passed to each thread). So Python printed the UserID of the for cycle instead of the "individual" userid passed to the different functions. It was only a logging problem, the program sent messages correctly.
I have a for that loops through the elements of a user's list. Each iteration, I would like to start a separate background thread to send a message to that user. By saying "send a message" I mean a simple POST request made using the requests Python lib. At the end of the thread, an output on the console is written. Every 24 requests (so every 24 threads) the app needs to stop for about a second.
Success = 0
Bounces = 0
def SendMessage(botid, token, userid, messageid, tag):
global Success
global Bounces
try:
payload = {...}
r = requests.post("...", params=payload, headers=head, timeout=2)
#problem with request?
pjson = json.loads(r.text)
if r.status_code != 200:
log(str(r.status_code) + " " + pjson["result"] + " UserID: " + UserID + "; URL: " + "..." + BotID + "/users/" + UserID + "/send; Params: " + str(payload))
Bounces += 1
return
Success += 1
return
except requests.exceptions.Timeout:
#wait for connection to be available again!
while not conn_available():
print("... Waiting for a new connection...")
time.sleep(10)
log("Request timed out. UserID: " + UserID + "; URL: " + "..." + BotID + "/users/" + UserID + "/send; Params: " + str(payload))
Bounces += 1
except requests.exceptions.ConnectionError:
log("Unable to connect. UserID: " + UserID + "; URL: " + "..." + BotID + "/users/" + UserID + "/send; Params: " + str(payload))
Bounces += 1
except requests.exceptions.HTTPError:
log("Invalid request. UserID: " + UserID + "; URL: " + "..." + BotID + "/users/" + UserID + "/send; Params: " + str(payload))
Bounces += 1
except requests.exceptions.RequestException:
log("Invalid request. UserID: " + UserID + "; URL: " + "..." + BotID + "/users/" + UserID + "/send; Params: " + str(payload))
Bounces += 1
while True:
newMsgsReq = ""
try:
#Check for new messages
newMsgsReq = requests.get("...", timeout=2)
if newMsgsReq.text == "false":
#exit sub
time.sleep(2)
continue
except requests.exceptions.HTTPError as errh:
log("Request has failed: There was an error in the request: [" + str(errh) + "]")
time.sleep(2)
continue
except requests.exceptions.ConnectionError as errc:
log("Request has failed: check internet connection & retry: [" + str(errc) + "]")
time.sleep(2)
continue
except requests.exceptions.Timeout as errt:
log("Request has failed: check internet connection & retry: [" + str(errt) + "]")
time.sleep(2)
continue
except requests.exceptions.RequestException as err:
log("Request has failed: There was an error in the request: [" + str(err) + "]")
time.sleep(2)
continue
#we have a message!!!
#Extract BotID, Token, MessageID
msgInf = newMsgsReq.text.split("|")
MessageID = msgInf[0]
BotID = msgInf[1]
Token = msgInf[2]
Tag = msgInf[3]
del msgInf[0:4]
suc("New message found: " + str(MessageID))
suc("Total recipients: " + str(len(msgInf)))
#Begin send!
Cycles = 0
TotCycles = 0
#Loop through msgInf
for UserID in msgInf:
#Create the thread.
process = threading.Thread(target=SendMessage, args=[BotID, Token, UserID, MessageID, Tag])
process.start()
TotCycles += 1
pb.print_progress_bar(TotCycles)
Cycles += 1
if Cycles == 24:
time.sleep(1)
Cycles = 0
suc("Message " + str(MessageID) + " sent successfully (" + str(Success) + " success, " + str(Bounces) + " bounces")
Success = 0
Bounces = 0
time.sleep(3)
Let's say my user list is:
{1, 2, 3, 4, ..., 24, 25, ...}. I expect my application to output:
1. Message 1 sent successfully...
2. Message 2 sent successfully...
...
24. Message 24 sent successfully.
Instead, I am getting this output:
1. Message 1 sent successfully.
2. Message 1 sent successfully.
...
24. Message 1 sent successfully.
So all the 24 outputs are related to the first of the 24 ids. It seems like the for loop does not proceed...

This prints the incremented counter without any trouble so I think you may need to provide all of the code and some sample input.
import threading
import time
def SendMessage(userid):
print(userid)
while True:
cycles = 1
for user_id in [1, 2, 3]:
process = threading.Thread(target=SendMessage, args=[user_id])
process.start()
cycles += 1
if cycles == 24:
time.sleep(1)
cycles = 0
time.sleep(3)
Run it on repl.it

Python request stacks

Hi i try to request some json, all is ok, but sometimes request goes wrong, may be site stack or closing connection for me, cause i am requesting json every 15 min.
Code here:
def request_coins(config):
coins = None
print(str(datetime.now()) + ' - REQUESTING Coins')
while coins is None:
try:
coins = (requests.get(url=str(config['UrlPath']['url']) + str(config['UrlPath']['userrates'])))
except:
print(str(datetime.now()) + " - Site didn't respond. Reconnecting in 10 sec")
time.sleep(10)
if coins is not None:
coins = coins.json()['coins']
print(str(datetime.now()) + ' - Coins received correctly')
return coins
Problems is that try - catch don't work in my case. Request stack in print(str(datetime.now()) + ' - REQUESTING Coins') and i don't recieve any exception and message to cmd
print(str(datetime.now()) + " - Site didn't respond. Reconnecting in 10 sec")
As I understood code stacks at try:
coins = (requests.get(url=str(config['UrlPath']['url']) +
str(config['UrlPath']['userrates'])))
and don't go even to except:
Any idea how to fix?

Just need to add timeout to get request

save_screenshot cannot work properly in for loop

medium_timeout = Data.medium_timeout
#click multiple buttons
#i = 0
for steps in buttonData:
if steps == "wait":
time.sleep(2)
elif steps == "back":
driver.back()
else:
try:
#wait for element to show up
WebDriverWait(driver, medium_timeout).until(EC.presence_of_element_located((By.XPATH, steps)))
WebDriverWait(driver, medium_timeout).until(EC.visibility_of_element_located((By.XPATH, steps)))
#crashpicturename = "before" + "error" + str(i) + ".png"
#driver.save_screenshot(crashpicturename)
except TimeoutException:
logging.debug(str(datetime.datetime.now()) + " TimeoutException: Unable to locate " + steps)
return driver
#Input some data where necessary
if steps == "//input[#id = 'fulfill']":
driver.find_element_by_xpath("//div[#id= 'orderItemsContainer']//input").send_keys(str(randint(10000, 99999)))
logging.debug(str(datetime.datetime.now()) + " Clicking on " + steps)
try:
driver.find_element_by_xpath(steps).click()
logging.debug(str(datetime.datetime.now()) + " after Clicking on " + steps)
crashpicturename = "after" + "error" + str(i) + ".png"
driver.save_screenshot(crashpicturename)
i += 1
except Exception as e:
logging.debug(str(datetime.datetime.now()) + format(e))
time.sleep(10)
driver.find_element_by_xpath(steps).click()
return driver
During this testing, I plan to click "user" first, then click "edit", then click"change email" to test these steps work normally, at the same time, save screenshots after every step. But it seems save_screenshot for PhantomJS can not work normally. For example, the screenshot after clicking "user" has the same screenshot after clicking "change email". IT is messy.