I'm trying to capture all network logs using seleniumwire. When chromedriver is in normal mode, it is able to capture all requests. But when it is in headless mode, it is not capturing all requests.
I tried adding sleep(10), assert driver.last_request.response.status_code == 200
but neither helped.
Since seleniumwire is not that popular, I'm adding a sample guide below in the hope of getting people with knowledge of selenium to try a hand to help me fix the problem.
Working with seleniumwire
Installing seleniumwire
pip install seleniumwire
Sample script:
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Chrome driver
driver = webdriver.Chrome()
# Go to the YouTube homepage.
driver.get('https://www.youtube.com')
# Access requests via the `requests` attribute
for request in driver.requests:
if request.response:
print(
request.path,
request.response.status_code,
request.response.headers['Content-Type']
)
try to capture all requests
options = {
'ignore_http_methods': [] # Capture all requests, including OPTIONS requests
}
driver = webdriver.Chrome("C:\chromedriver.exe",seleniumwire_options=options)
In default it ignores OPTIONS method
When chrome browser is opened by selenium, it uses it's own profile rather than the default one present. Try using custom profile, for chrome you can use ChromeOptions class use a custom profile and try.
Related
When I run this code, the page opens and closes. I can't reach the page. Everything is in the latest version. I am a Windows user.
I was trying to open Instagram page with this code.
enter image description here
I tried to open the instagram site with this code and the site was closed as soon as it was opened.
Im not familiar with selenium but I know for a fact that
get() just gets the HTML code of the website and does not display it
You have to add the below Chrome Option:
options.add_experimental_option("detach", True)
Full code:
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_experimental_option("detach", True)
driver = webdriver.Chrome(service=Service(chrom_driver_pat), options=options)
driver.get(<url>)
Next time, don't post the image, post the code and explain your issue clearly.
First of all, please, do not use images when your want a code review or help, it is more easier to help with raw code.
Selenium now uses Service method to run webdriver.
To run what you want, you need to fix some parts of your code, like this:
from selenium import webdriver
# in my case, i'm using selenium==4.5.0, for this version, is necessary to
# use selenium.webdriver.chrome.service to use driver path
from selenium.webdriver.chrome.service import Service as ChromeService
# For Windows path, use r"C:\..." when you will use \
# Remember to set the executable file too
chrome_driver_path = r"C:\pythondriver\chromedriver\chromedriver.exe"
# you can use "universal" path too, like linux, using / instead of \
# chrome_driver_path = "C:/pythondriver/chromedriver/chromedriver.exe"
# instead using:
# webdriver.Chrome()
# driver = webdriver.Chrome(chrome_driver_path)
# use this:
chrome_service = ChromeService(chrome_driver_path)
driver = webdriver.Chrome(service=chrome_service)
url = "https://instagram.com"
driver.get(url)
I tested using the configurations below and worked for me:
Chromedriver - 108.0.5359.71
Chrome - 108.0.5359.125 - 64 bits
If you need more help using Chrome Service, try look the Selenium Docs
I'm trying to write several tests using selenium, but I'm seeing the following strange behavior.
When I run the tests like this:
from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
options = FirefoxOptions()
service = Service()
brow = Firefox(service=service, options=options)
brow.execute("get", {'url': 'https://python.org'})
I get the result I expected, the python.org website is opened in Firefox browser.
But if I make a mistake in URL, I'm getting the following error:
from selenium.webdriver import Firefox, FirefoxOptions
from selenium.webdriver.firefox.service import Service
options = FirefoxOptions()
service = Service()
brow = Firefox(service=service, options=options)
brow.execute("get", {'url': 'qwerty'})
selenium.common.exceptions.InvalidArgumentException: Message: Malformed URL: URL constructor: qwerty is not a valid URL.
Stacktrace:
WebDriverError#chrome://remote/content/shared/webdriver/Errors.jsm:186:5
InvalidArgumentError#chrome://remote/content/shared/webdriver/Errors.jsm:315:5
GeckoDriver.prototype.navigateTo#chrome://remote/content/marionette/driver.js:804:11
I just want to understand why I see here WebDriverError#chrome, and not WebDriverError#firefox or something like that.
Is this a bug, or am I doing something wrong?
These error messages...
WebDriverError#chrome://remote/content/shared/webdriver/Errors.jsm:186:5
InvalidArgumentError#chrome://remote/content/shared/webdriver/Errors.jsm:315:5
GeckoDriver.prototype.navigateTo#chrome://remote/content/marionette/driver.js:804:11
containing the phrase #chrome may leave an impression of a strange behavior while using GeckoDriver and firefox combo.
However, as per #AutomatedTester's comment in the GitHub discussion Selenium 3.4.0-GeckoDriver 0.17.0 : GeckoDriver producing logs through Chromium/Chrome modules #787:
These errors are nothing to worry about. Mozilla uses different open source projects to build Firefox for different reasons. It showing Chrome errors means nothing in the big picture.
So you can ignore them safely.
I have been trying to create a script to login and post an ad. However when directing selenium either the "post an ad" page or the "login" page, the URL doesn't load, and gives me a solid white screen. However when I direct the URL to the homepage or any other page, other than the ones already listed, the website loads just fine.
My problem solving:
I have read that my IP might have got blocked, however when I change my public IP address through my router, the problem persists.
I am currently unsure if the source of the problem stands from my IP or the possibly the website's scripts are only meant to detect and block scripts on certain pages.
I am also using chromedriver version 81.0.4044.69 and Chrome version Version 81.0.4044.129 (Official Build) (64-bit)
Here is my code:
from selenium import webdriver
browser = webdriver.Chrome()
browser.get('https://www.kijiji.ca/t-login.html?targetUrl=L3Atc2VsZWN0LWNhdGVnb3J5Lmh0bWw/XnIrUFRKMS9oU1cxc29PdXAxbjUveFE9PQ--')
The result, and no errors
Blank White Screen Found When Running The Program
Try faking the user-agent (some sites really are picky in regards to that) using this module
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
options = Options()
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options.add_argument(f'--user-agent={userAgent}')
driver = webdriver.Chrome(options=options)
driver.get("https://www.google.co.in")
driver.quit()
geckodriver does launch Firefox but firefox doesn't get url. Please see and point what's wrong with my function. would be great help as im very new to selenium and python
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
def Login(SiteUrl):
driver = webdriver.Firefox()
driver.get(SiteUrl)
if __name__ =="__main__":
url = "www.google.com"
Login(url)
There could be multiple reasons -
Try including HTTP protocol in the url i.e. - http://www.google.com
You might be behind a proxy server. See this SO question -> Selenium WebDriver: Firefox starts, but does not open the URL & Selenium WebDriver.get(url) does not open the URL
Your versions of the driver and the browser does not match.
That simply means either your libs are old or Gecko binary is old. New gecko-libs are available now.
Download it from :
https://github.com/mozilla/geckodriver/releases
Download new version of selenium python libs from below URL:
https://pypi.python.org/pypi/selenium
Another thing as Martin point out
Add the protocol like http or https before passing URL
So use URL as :
url = "https://www.google.com"
I use a simple webdriver phantomjs script to update some adverts on preloved.co.uk. This script worked great until recently, but then started failing with the "Click submitted but load failed" error after the login link was clicked. In accordance with this I updated my version of phantomjs to latest stable, 1.9.7 following the guide here. However, now the login click does not seem to register either, and the page does not reload.
The first step is simply getting to login form page.
from selenium import webdriver
br = webdriver.PhantomJS(service_log_path='/path/to/logfile.log')
url = "http://www.preloved.co.uk"
br.get(url)
# Go to login page
login_button = br.find_element_by_xpath('//div[#id="header-status-login"]/a')
login_button.click()
Normally (and if you replace the browser line with br = webdriver.Firefox() for example), this results in reloading to login page, and the script proceeds from there, but now it appears the click does not load the new page at all and br.current_url is still 'http://www.preloved.co.uk/'
Why doesn't this load work?
Even if I extract the href and do an explicit GET it doesn't seem to follow and reload:
newurl=login_button.get_attribute('href')
br.get(newurl)
br.current_url is still 'http://www.preloved.co.uk/'.
The login page is secured through https. Recently the POODLE vulnerability forced websites to move away from SSLv3 for https, but since PhantomJS uses SSLv3 per default the login page doesn't load. See also this answer.
This can be fixed by passing --ssl-protocol=tlsv1 or --ssl-protocol=any to PhantomJS or upgrading PhantomJS to at least version 1.9.8. It seems that the service_args argument could be used for that in the python bindings for Selenium.
It looks like in the current official implementation the service_args cannot be passed from WebDriver to the Service in PhantomJS. You can sub-class it.
from selenium import webdriver
from selenium.webdriver.phantomjs.service import Service
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.remote.webdriver import WebDriver as RemoteWebDriver
class PhantomJSService(webdriver.PhantomJS):
def __init__(self, executable_path="phantomjs", port=0,
desired_capabilities=DesiredCapabilities.PHANTOMJS,
service_args=None, service_log_path=None):
self.service = Service(executable_path,
port=port, service_args=service_args,
log_path=service_log_path)
self.service.start()
try:
RemoteWebDriver.__init__(self,
command_executor=self.service.service_url,
desired_capabilities=desired_capabilities)
except:
self.quit()
raise
It seems that this webdriver fork contains the necessary arguments to set those options.