I have created a browser scraping script which sends a message on WhatsApp web using selenium in python but yesterday noticed a that its sending half message or not sending messages. Debugged it and found that the browser window must be active to send messages my send message code as below.
def send_message(msg):
whatsapp_msg = driver.find_element_by_class_name(send_messageClass)
for part in msg.split('\n'):
whatsapp_msg.send_keys(part)
ActionChains(driver).key_down(Keys.SHIFT).key_down(Keys.ENTER).key_up(Keys.SHIFT).key_up(Keys.ENTER).perform()
time.sleep(1)
ActionChains(driver).send_keys(Keys.RETURN).perform()
time.sleep(1)
find_element_by_class_name simply retrieves the element from the DOM. It does not ensure, if it is visible.
For this use an explicit wait in conjunction with visibility of the element as expected condition:
selenium.webdriver.support.expected_conditions.visibility_of(element)
This will wait for the element to be visible until the specified timeout is reached. Here is an example with timeout of 60 seconds:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EXP_CON
...
wait = WebDriverWait(driver, 60)
whatsapp_msg = driver.find_element_by_class_name(send_messageClass)
visible_whatsapp_msg = wait.until(EXP_CON.visibility_of(whatsapp_msg))
Related
I am using Selenium to log into a HikVision IP camera web client and I am experiencing an issue. The automated browser seems to successfully log in, as it loads the next page, but when I return page_source(), it shows that the webdriver is still stuck on the log in page. I have tried implementing wait/until and implicit waits for the web page to load but it does not seem to be a loading issue, as no matter how long I wait, I get the same problem.
Here is a code snippet showing how I log in:
user = driver.find_element_by_id("username")
password = driver.find_element_by_id("password")
# log in
user.clear()
user.send_keys("admin")
password.clear()
password.send_keys("Snipr123")
driver.find_element_by_css_selector(".btn.btn-primary.login-btn").click()
The .clear() was just to get rid of the preloaded text that was giving me issues.
After the login click() the automated browser successfully loads the web client, but the webdriver doesn't, since page_source() still returns the log in page.
Any ideas on what could be going wrong would be greatly appreciated.
Once you login using the line of code:
driver.find_element_by_css_selector(".btn.btn-primary.login-btn").click()
Yneed to induce WebDriverWait for the visibility_of_element_located() of any of the visible element within the DOM Tree and then extract the page source as follows:
driver.find_element_by_css_selector(".btn.btn-primary.login-btn").click()
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CLASS_NAME, "table")))
print(driver.page_source)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Is it possible to make a program in python in which the program will look for a message on WhatsApp sent by a particular person on a group(which I have pinned for convenience) at a particular time and then it will print the message sent by that person?
Well if you don't understand my question let me give you an example:
Suppose there is a group named ABCD which is pinned on WhatsApp web(which means it is at the top)
It consists of 4 people - A,B,C,D
I want the program to print the message sent by C at time 13:05 in that group
Is this thing possible on python?
I can use any module like selenium or even pyautogui
Yes, it is possible through Selenium without using any kind of API.
First we will import required modules:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
import time
Now we will launch Whatsapp Web:
driver = webdriver.Chrome()
driver.get("https://web.whatsapp.com")
You can use other browsers, for this example we will be using Chrome.
Now we will locate the Search or Start New Chat button using XPath and add a webdriver wait for it:
search_button = WebDriverWait(driver,50).until(lambda driver: driver.find_element_by_xpath("//input[#title='Search or start new chat']"))
Now we will click it and add a little sleep:
search_button.click()
time.sleep(2)
Now we will Send the name of the Person to contact into the text area:
search_button.send_keys("A")
time.sleep(2)
Now we will send a message
input_box = driver.find_element_by_xpath(r'//div[#class="_2S1VP copyable-text selectable-text"][#contenteditable="true"][#data-tab="1"]')
time.sleep(2)
input_box.send_keys("Hello" + Keys.ENTER)
time.sleep(2)
You can using Chrome inspect more and do more things.
I have a bot that I'm testing and it works for the most part, but every once in a while when it navigates to a new page, Chrome throws a "Reload Page?" alert and it stops the bot. How can I add a check in the bot to look for this alert, and if it's there click the "Reload" button on the alert?
In my code I have
options.add_argument("--disable-popup-blocking")
and
driver = webdriver.Chrome(chrome_options=options, executable_path="chromedriver.exe")
but it still happens every once in a while. Any advice?
You can use driver.switch_to_alert to handle this case.
I would also invoke WebDriverWait on the alert itself, to avoid the NoSuchAlert exception:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
def refresh_with_alert(driver):
# wrap this in try / except so the whole code does not fail if alert is not present
try:
# attempt to refresh
driver.refresh()
# wait until alert is present
WebDriverWait(driver, 5).until(EC.alert_is_present())
# switch to alert and accept it
driver.switch_to.alert.accept()
except TimeoutException:
print("No alert was present.")
Now, you can call like this:
# refreshes the page and handles refresh alert if it appears
refresh_with_alert(driver)
The above code will wait up to 5 seconds to check if an alert is present -- this can be shortened based on your code needs. If the alert is not present, the TimeoutException will be hit in the except block. We simply print a statement that the alert does not exist, and the code will move on without failing.
If the alert is present, then the code will accept the alert to close it out.
I've written a script in Python in association with selenium to click on each of the signs available in a map. However, when I execute my script, it throws timeout exception error upon reaching this line wait.until(EC.staleness_of(item)).
Before hitting that line, the script should have clicked once but It could not? How can I click on all the signs in that map cyclically?
This is the site link.
This is my code so far (perhaps, I'm trying with the wrong selectors):
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link = "https://www.findapetwash.com/"
driver = webdriver.Chrome()
driver.get(link)
wait = WebDriverWait(driver, 15)
for item in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "#map .gm-style"))):
item.click()
wait.until(EC.staleness_of(item))
driver.quit()
Signs visible on that map like:
Post script: I know that this is their API https://www.findapetwash.com/api/locations/getAll/ using which I can get the JSON content but I would like to stick to the Selenium way. Thanks.
I know you wrote you don't want to use the API but using Selenium to get the locations from the map markers seems a bit overkill for this, instead, why not making a call to their Web service using requests and parse the returned json?
Here is a working script:
import requests
import json
api_url='https://www.findapetwash.com/api/locations/getAll/'
class Location:
def __init__(self, json):
self.id=json['id']
self.user_id=json['user_id']
self.name=json['name']
self.address=json['address']
self.zipcode=json['zipcode']
self.lat=json['lat']
self.lng=json['lng']
self.price_range=json['price_range']
self.photo='https://www.findapetwash.com' + json['photo']
def get_locations():
locations = []
response = requests.get(api_url)
if response.ok:
result_json = json.loads(response.text)
for location_json in result_json['locations']:
locations.append(Location(location_json))
return locations
else:
print('Error loading locations')
return False
if __name__ == '__main__':
locations = get_locations()
for l in locations:
print(l.name)
Selenium
If you still want to go the Selenium way, instead of waiting until all the elements are loaded, you could just halt the script for some seconds or even a minute to make sure everything is loaded, this should fix the timeout exception:
import time
driver.get(link)
# Wait 20 seconds
time.sleep(20)
For other possible workarounds, see the accepted answer here: Make Selenium wait 10 seconds
You can click one by one using Selenium if, for some reasons, you cannot use API. Also it is possible to extract information for each sign without clicking on them with Selenium.
Here code to click one by one:
signs = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "li.marker.marker--list")))
for sign in signs:
driver.execute_script("arguments[0].click();", sign)
#do something
Try also without wait, probably will work.
I am trying to create a bot which will fill up the signup forms for a particular website.
Website Details - http://hacknyu.org/signup
Code:
from selenium import webdriver
class HackNNYU(object):
first_name = 'input[ng-model="credentials.first_name"]'
last_name = 'input[ng-model="credentials.last_name"]'
email = '.col-sm-12>input[ng-model="credentials.email"]'
password = '.col-sm-12>input[ng-model="credentials.password"]'
agree_checkbox = '.ng-binding>input[ng-model="checkModel"]'
sign_up_button = 'div>button[type="submit"]'
accept_button = 'button[ng-click="positive()"]'
def fill_up_hack_nyu():
driver = webdriver.Firefox()
driver.get('http://hacknyu.org/signup')
driver.find_element_by_css_selector(HackNNYU.first_name).send_keys('Friday')
driver.find_element_by_css_selector(HackNNYU.last_name).send_keys('Night')
driver.execute_script("window.scrollTo(0, 300);")
driver.find_element_by_css_selector(HackNNYU.email).send_keys('ade347#gmail.edu')
driver.find_element_by_css_selector(HackNNYU.password).send_keys('123456')
driver.find_element_by_css_selector(HackNNYU.agree_checkbox).click()
driver.find_element_by_css_selector(HackNNYU.accept_button).click()
# driver.execute_script("window.scrollTo(0, 400);")
driver.find_element_by_css_selector(HackNNYU.sign_up_button).click()
fill_up_hack_nyu()
Problem
driver.find_element_by_css_selector(HackNNYU.sign_up_button).click()
The main problem is in this line. Manually when you click on signup button it is working fine but when I run this program, I can see it is clicking on signup button but nothing is happening after that. Could anyone help me why this is not working? I am just trying to register for an event using a bot. I will highly appreciate any help you can provide.
Error
Sometimes I always receive this error
selenium.common.exceptions.WebDriverException: Message: Element is not clickable at point (796.4000244140625, 45.399993896484375). Other element would receive the click
The page has a banner at the top which makes the automatic scrolling hide the submit button. To overcome this issue, you can define the scrolling behaviour to scroll at the bottom instead. Moreover it seems that your script is no correctly clicking on the checkbox that should display the terms popup.
Here is a working script to create a new account on hacknyu:
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# set the scrolling behavior to down
DesiredCapabilities.FIREFOX["elementScrollBehavior"] = 1
driver = webdriver.Firefox()
wait = WebDriverWait(driver, 10)
# load the page
driver.get("http://hacknyu.org/signup")
# get the form element
form = driver.find_element_by_css_selector("form[name='signupForm']")
# fill the fields
form.find_element_by_css_selector("input[name='firstName']").send_keys("myfirstname")
form.find_element_by_css_selector("input[name='lastName']").send_keys("mylastname")
form.find_element_by_css_selector("input[name='email']").send_keys("na#na.na")
form.find_element_by_css_selector("input[name='password']").send_keys("mypassword")
# click and accept terms
form.find_element_by_xpath("//input[#name='terms']/..").click()
wait.until(EC.presence_of_element_located((By.XPATH, "//button[.='Accept']"))).click()
wait.until_not(EC.presence_of_element_located((By.CSS_SELECTOR, ".modal")))
# click on submit
form.find_element_by_css_selector("button[type='submit']").click()
It is probably timing issue. You can use explicit wait to make sure the element is clickable
wait = WebDriverWait(driver, 10)
wait.until(expected_conditions.element_to_be_clickable((By.CSS_SELECTOR, HackNNYU.sign_up_button))).click()
This will wait up to 10 seconds for the button to be clickable before clicking on it.
Mostly providing wait or sleep before click will helps you simulate click correctly. Some situations click by using Actions also helps me. In Java
Actions moveAndClick=new Actions(driver);
moveAndClick.moveToElement(driver.findElement(By.cssSelector("HackNNYU.sign_up_button"))).click().build().perform();
and one more way is to use javascript executor.
WebElement element = driver.findElement(By.cssSelector("HackNNYU.sign_up_button"));
JavascriptExecutor executor = (JavascriptExecutor)driver;
executor.executeScript("arguments[0].click();", element);
Thank You,
Murali