I use a simple webdriver phantomjs script to update some adverts on preloved.co.uk. This script worked great until recently, but then started failing with the "Click submitted but load failed" error after the login link was clicked. In accordance with this I updated my version of phantomjs to latest stable, 1.9.7 following the guide here. However, now the login click does not seem to register either, and the page does not reload.
The first step is simply getting to login form page.
from selenium import webdriver
br = webdriver.PhantomJS(service_log_path='/path/to/logfile.log')
url = "http://www.preloved.co.uk"
br.get(url)
# Go to login page
login_button = br.find_element_by_xpath('//div[#id="header-status-login"]/a')
login_button.click()
Normally (and if you replace the browser line with br = webdriver.Firefox() for example), this results in reloading to login page, and the script proceeds from there, but now it appears the click does not load the new page at all and br.current_url is still 'http://www.preloved.co.uk/'
Why doesn't this load work?
Even if I extract the href and do an explicit GET it doesn't seem to follow and reload:
newurl=login_button.get_attribute('href')
br.get(newurl)
br.current_url is still 'http://www.preloved.co.uk/'.
The login page is secured through https. Recently the POODLE vulnerability forced websites to move away from SSLv3 for https, but since PhantomJS uses SSLv3 per default the login page doesn't load. See also this answer.
This can be fixed by passing --ssl-protocol=tlsv1 or --ssl-protocol=any to PhantomJS or upgrading PhantomJS to at least version 1.9.8. It seems that the service_args argument could be used for that in the python bindings for Selenium.
It looks like in the current official implementation the service_args cannot be passed from WebDriver to the Service in PhantomJS. You can sub-class it.
from selenium import webdriver
from selenium.webdriver.phantomjs.service import Service
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.remote.webdriver import WebDriver as RemoteWebDriver
class PhantomJSService(webdriver.PhantomJS):
def __init__(self, executable_path="phantomjs", port=0,
desired_capabilities=DesiredCapabilities.PHANTOMJS,
service_args=None, service_log_path=None):
self.service = Service(executable_path,
port=port, service_args=service_args,
log_path=service_log_path)
self.service.start()
try:
RemoteWebDriver.__init__(self,
command_executor=self.service.service_url,
desired_capabilities=desired_capabilities)
except:
self.quit()
raise
It seems that this webdriver fork contains the necessary arguments to set those options.
Related
Is selenium only for testing?
I created a script to log in to canvas, a website that my uni uses for class material
however, it seems that it only logs in on the browser generated by the driver, and I will still have to manually log in on the actual browser.
Is there a way for me to make it so that I won't have to log in on the actual browser after running my script?
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time
PATH = r"C:the path \msedgedriver.exe"
driver = webdriver.Edge(PATH)
driver.get("the website")
driver.maximize_window()
#sign in page
user = driver.find_element(By.ID,"username")
user.send_keys("username")
pw = driver.find_element(By.ID,"password")
pw.send_keys("password")
pw.send_keys(Keys.RETURN)
driver.implicitly_wait(3)
#authentification
driver.switch_to.frame(driver.find_element(By.XPATH,"//iframe[#id='duo_iframe']"))
remember_me = driver.find_element(By.XPATH, "//input[#type='checkbox']")
remember_me.click()
duo = driver.find_element(By.XPATH, '//button[text()="Send Me a Push "]')
duo.click()
As per Selenium's Homepage
Selenium automates browsers. That's it! What you do with that power is
entirely up to you.
Primarily it is for automating web applications for testing purposes,
but is certainly not limited to just that.
Boring web-based administration tasks can (and should) also be
automated as well.
No, you won't be able to reconnect to the already opened Browsing Context.
Even if you are able to extract the ChromeDriver and ChromeSession attributes e.g. Session ID, Cookies, UserAgent and other session attributes from the already initiated ChromeDriver and Chrome Browsing Session still you won't be able to change the set of attributes of the ChromeDriver.
A cleaner way would be to span a new set of ChromeDriver and Chrome Browser instance with the desired set of configurations.
I'm new to Python. I'm trying to do automation by opening a login page in Selenium.
from selenium import webdriver
browser = webdriver.Chrome(executable_path='chromedriver')
I tried to test some sites like - 'https://www.google.com/',etc. which is working perfectly fine.
url = 'https://www.google.com/'
browser.get(url)
I'm trying to open below url,
url = 'https://yesonline.yesbank.co.in/index.html?module=login'
browser.get(url)
I got the following error in selenium browser while the url is working fine without selenium.
Access Denied
You don't have permission to access
"http://yesonline.yesbank.co.in/index.html?" on this server.
Reference
#18.ef87d317.1625646692.41fe4bc0
But when I'm trying to just open the base url, it is opening but the site gets loads partially and keep showing loading.
url = 'https://yesonline.yesbank.co.in'
browser.get(url)
I feel like I am missing out something while opening the login url which I'm not able to get what exactly.
I also tried changing the webdriver i.e with Firefox.
url = 'https://yesonline.yesbank.co.in'
firefox_browser = webdriver.Firefox()
And guess what, it was opening!
But as soon as I'm trying to get the login page (even by manually using the mouse and clicking login page).
url = 'https://yesonline.yesbank.co.in/index.html?module=login'
firefox_browser.get(url)
'firefox_browser' is getting closed with an session reset error.
Can someone help me how to open secure sites in selenium. Or is there any other way to get it done.
It's finally working with chrome-driver by adding some arguments to it.
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('disable-infobars')
options.add_argument('--disable-extensions')
options.add_argument('--disable-blink-features=AutomationControlled')
browser = webdriver.Chrome(executable_path='chromedriver', options = options)
I have been trying to create a script to login and post an ad. However when directing selenium either the "post an ad" page or the "login" page, the URL doesn't load, and gives me a solid white screen. However when I direct the URL to the homepage or any other page, other than the ones already listed, the website loads just fine.
My problem solving:
I have read that my IP might have got blocked, however when I change my public IP address through my router, the problem persists.
I am currently unsure if the source of the problem stands from my IP or the possibly the website's scripts are only meant to detect and block scripts on certain pages.
I am also using chromedriver version 81.0.4044.69 and Chrome version Version 81.0.4044.129 (Official Build) (64-bit)
Here is my code:
from selenium import webdriver
browser = webdriver.Chrome()
browser.get('https://www.kijiji.ca/t-login.html?targetUrl=L3Atc2VsZWN0LWNhdGVnb3J5Lmh0bWw/XnIrUFRKMS9oU1cxc29PdXAxbjUveFE9PQ--')
The result, and no errors
Blank White Screen Found When Running The Program
Try faking the user-agent (some sites really are picky in regards to that) using this module
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
options = Options()
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options.add_argument(f'--user-agent={userAgent}')
driver = webdriver.Chrome(options=options)
driver.get("https://www.google.co.in")
driver.quit()
So I'm trying to make a python script using selenium and bs4 to automatically buy shoes for from adidas.com. It's just that whenever selenium browser starts the adidas site it shows a popup concerning cookies and privacy. I can't click on the accept button using selenium(can't find the element to click on) and I've tried starting the selenium browser with my firefox profile containing cookies and what not. But it still shows that damn popup and it's stopping the script.
Is there anyway to fix this?
Tried starting selenium with firefox cookies like this:
ffprofile = webdriver.FirefoxProfile(r'C:\Users\chico\AppData\Local\Mozilla\Firefox\Profiles\e5108gza.default')
driver = webdriver.Firefox(firefox_profile=ffprofile)
driver.get('https://www.adidas.nl/on/demandware.store/Sites-adidas-NL-Site/nl_NL/MyAccount-CreateOrLogin')
driver.find_element_by_xpath("//input[#id='dwfrm_login_username']").send_keys('email')
driver.find_element_by_xpath("//input[#id='dwfrm_login_password']").send_keys('password')
driver.find_element_by_xpath("//button[#value='Inloggen']").click()
The popup from the adidas site just keeps poping up and stopping my script from continuing. Sometimes it will fill in email and password before being stopped, sometimes it will be stopped before that.
I ran into this same issue recently, and it was SO frustrating, because it can't be handled with a "switch to alert" and "accept" it approach sometimes.
Here's how I got around my issue. First, and this was most important, you need to "inspect" the page you are writing selenium code for in a way that the cookie acceptance popup will show up.
For me, I was using the Chrome driver, and so for inspecting the page, I had to run incognito, so that the page would not know I had already accepted cookies. Use the method for Firefox that will achieve the same effect.
Then you can find the element ID (or class name) for the acceptance button and add a click to the end of your command. My code to do this looks as follows:
from selenium import webdriver
import time
URL = "http:some_site.com"
driver = webdriver.Chrome()
driver.get(URL)
time.sleep(5)
driver.find_element_by_id(the_id_name_for_cookie_acceptance_as_a_string).click()
# Do more stuff ...
Now, if you run into a cookie dialog that won't act as an alert, you can handle it.
I tried the code you posted and didn't receive any alert pop up.But if there is a Alert windows it needs to be handled by switching to it and accepting it
from selenium import webdriver
driver = webdriver.Firefox()
driver.set_page_load_timeout(30)
driver.get('https://www.adidas.nl/on/demandware.store/Sites-adidas-NL-Site/nl_NL/MyAccount-CreateOrLogin')
try:
WebDriverWait(browser, 3).until(EC.alert_is_present(),'Timed out waiting for alert popup')
alert = driver.switch_to.alert
alert.accept()
except:
print "no alert"
driver.find_element_by_xpath("//input[#id='dwfrm_login_username']").send_keys('email')
driver.find_element_by_xpath("//input[#id='dwfrm_login_password']").send_keys('password')
driver.find_element_by_xpath("//button[#value='Inloggen']").click()
I'm trying to capture all network logs using seleniumwire. When chromedriver is in normal mode, it is able to capture all requests. But when it is in headless mode, it is not capturing all requests.
I tried adding sleep(10), assert driver.last_request.response.status_code == 200
but neither helped.
Since seleniumwire is not that popular, I'm adding a sample guide below in the hope of getting people with knowledge of selenium to try a hand to help me fix the problem.
Working with seleniumwire
Installing seleniumwire
pip install seleniumwire
Sample script:
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Chrome driver
driver = webdriver.Chrome()
# Go to the YouTube homepage.
driver.get('https://www.youtube.com')
# Access requests via the `requests` attribute
for request in driver.requests:
if request.response:
print(
request.path,
request.response.status_code,
request.response.headers['Content-Type']
)
try to capture all requests
options = {
'ignore_http_methods': [] # Capture all requests, including OPTIONS requests
}
driver = webdriver.Chrome("C:\chromedriver.exe",seleniumwire_options=options)
In default it ignores OPTIONS method
When chrome browser is opened by selenium, it uses it's own profile rather than the default one present. Try using custom profile, for chrome you can use ChromeOptions class use a custom profile and try.