I'm unable lo load my Firefox profile on the Selenium Webdriver.
i have tried several ways commented here on SO, but non did the trick. Either i loose connection to the driver or the profile does not get loaded.
this is what i have so far:
from selenium import webdriver
options = Options()
options.headless = False
fp = webdriver.FirefoxProfile(r'C:\\Users\\xxXX\\AppData\\Local\\Mozilla\\Firefox\\Profiles\\y73h6ogb.default-release')
driver = webdriver.Firefox(fp, options=options, executable_path='geckodriver/geckodriver.exe')
driver.get(URL)
According to the documentation you can load a profile like this:
from selenium.webdriver import Firefox, FirefoxOptions, FirefoxProfile
options = FirefoxOptions()
options.headless = True
profile = FirefoxProfile("PATH_TO_PROFILE\\1234asdf.some_profile")
driver = Firefox(
firefox_profile=profile,
executable_path="PATH_TO_DRIVER\\geckodriver.exe",
options=options,
)
driver.get("https://python.org")
driver.close()
Or by adding the profile to the options:
...
options = FirefoxOptions()
options.headless = True
options.profile = FirefoxProfile("PATH_TO_PROFILE\\1234asdf.some_profile")
driver = Firefox(
executable_path="PATH_TO_DRIVER\\geckodriver.exe",
options=options,
)
...
By using webdriver for firefox, You will be able to do that!
Related
I have followed the change exact but it doesn't work, are there any other bugs?
https://www.selenium.dev/blog/2023/headless-is-going-away/
import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
options = webdriver.ChromeOptions()
options.headless = True
options.page_load_strategy = 'none'
chrome_path = 'chromedriver.exe'
chrome_service = Service(chrome_path)
driver = Chrome(options=options, service=chrome_service)
driver.implicitly_wait(5)
url= "https://hk.centanet.com/findproperty/list/transaction/%E6%84%89%E6%99%AF%E6%96%B0%E5%9F%8E_3- DMHSZHHRHD?q=TiDxvVGMUUeutVzA0g1JlQ"
driver.get(url)
time.sleep(10)
contents = driver.find_element(By.CSS_SELECTOR,"div[class*='bx--structured-list-tbody']")
properties = contents.find_elements(By.CSS_SELECTOR,"div[class*='bx--structured-list-row']")
def extract_data(element):
columns = element.find_elements(By.CSS_SELECTOR,"div[class*='bx--structured-list-td']")
Date = columns[0].text
Dev = columns[1].text
Price = columns[3].text
RiseBox = columns[4].text
Area = columns[5].text
return{
"Date": Date,
"Development": Dev,
"Consideration": Price,
"Change": RiseBox,
"Area": Area
}
data = []
for property in properties:
extracted_data = extract_data(property)
data.append(extracted_data)
df = pd.DataFrame(data)
df.to_csv("result.csv", index=False)
Comment was as follows:
Selenium.py:10: DeprecationWarning: headless property is deprecated, instead use add_argument('--headless') or add_argument('--headless=new')
options.headless = True
I tried the changes suggested by https://www.selenium.dev/blog/2023/headless-is-going-away/
but it doesn't work
This warning message...
DeprecationWarning: headless property is deprecated, instead use add_argument('--headless') or add_argument('--headless=new') options.headless = True
...is inline with the Selenium Blog post with the heading Headless is Going Away!
Details
Chromium team have released the Native Headless mode which is now officially the new Headless mode. This functionality is available with:
Chromium v109.0.5400.0
ChromeDriver v109.0.5414.25
Selenium v4.8.0
The new syntax requires --headless=new to be passed as an argument as follows:
Java:
ChromeOptions options = new ChromeOptions();
options.addArguments("--headless=new");
WebDriver driver = new ChromeDriver(options);
driver.get("https://selenium.dev);
driver.quit();
Python:
options = ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
driver.get('http://selenium.dev')
driver.quit()
Javascript:
let driver = await env
.builder()
.setChromeOptions(options.addArguments('--headless=new'))
.build();
await driver.get('https://selenium.dev');
await driver.quit();
CSharp:
var options = new ChromeOptions();
options.AddArgument("--headless=new");
var driver = new ChromeDriver(options);
driver.Navigate().GoToUrl("https://selenium.dev");
driver.Quit();
Ruby:
options = Selenium::WebDriver::Options.chrome(args: ['--headless=new'])
driver = Selenium::WebDriver.for :chrome, options: options
driver.get('https://selenium.dev')
driver.quit
Try this configuration:
# pip install selenium==4.8.0
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
options=Options()
options.add_argument("--headless=new")
options.add_experimental_option("excludeSwitches", ["enable-logging"]) #Optional
service=Service(f"path\chromebrowser.exe")
browser=webdriver.Chrome(service=service, options=options)
print(browser.name)
pythonseleniumwebdriveroptions
I'm trying to interact with the page "Your connection is not private".
The solution of using options.add_argument('--ignore-certificate-errors') is not helpful for two reasons:
I'm using an already open window.
Even if I was using a "selenium opened window" the script runs non stop, and the issue I'm trying to solve is when my browser disconnects from a splunk dashboard and I want it to automatically connect again(and it pops the private connection window).
How do I click on "Advanced" and then click on "Proceed to splunk_server (unsafe)?
For chrome:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-ssl-errors=yes')
options.add_argument('--ignore-certificate-errors')
driver = webdriver.Chrome(options=options)
If not work then this:
from selenium import webdriver
from selenium.webdriver import DesiredCapabilities
options = webdriver.ChromeOptions()
options.add_argument('--allow-insecure-localhost') # differ on driver version. can ignore.
caps = options.to_capabilities()
caps["acceptInsecureCerts"] = True
driver = webdriver.Chrome(desired_capabilities=caps)
For firefox:
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.accept_untrusted_certs = True
driver = webdriver.Firefox(firefox_profile=profile)
driver.get('https://cacert.org/')
driver.close()
If not work then this:
capabilities = webdriver.DesiredCapabilities().FIREFOX
capabilities['acceptSslCerts'] = True
driver = webdriver.Firefox(capabilities=capabilities)
driver.get('https://cacert.org/')
driver.close()
Above all worked for me!
This is how i handle this problem:
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.remote.CapabilityType;
ChromeOptions capability = new ChromeOptions();
capability.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
capability.setCapability(CapabilityType.ACCEPT_INSECURE_CERTS,true);
WebDriver driver = new ChromeDriver(capability);
This chrome option is the silver bullet for me:
chromeOptions.addArguments("--allow-running-insecure-content");
If you need more, Open chrome & paste this URL:
chrome://flags/
One will find all the options and their impact on the chrome.
Either of below 2 solutions worked for me using Python Chrome Selenium Webdriver:
from selenium import webdriver
from selenium.webdriver import DesiredCapabilities
capabilities = DesiredCapabilities.CHROME.copy()
capabilities["acceptInsecureCerts"] = True
driver = webdriver.Chrome(desired_capabilities=capabilities)
And accepted solution:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--ignore-ssl-errors=yes')
options.add_argument('--ignore-certificate-errors')
driver = webdriver.Chrome(options=options)
While navigating to the page, It is able to extract the data.
Code:
from selenium.webdriver.chrome.options import Options
options1 = Options()
options1.headless = True
driver = webdriver.Chrome(os.getcwd() +"/chromedriver",options = options1)
this is how I am getting the chrome driver
content = BeautifulSoup(driver.page_source,"html.parser")
this is how I am getting the content from the navigated page. While we are navigating to the page able to extract the data.
Can you try it like below?
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(os.getcwd() +"/chromedriver", chrome_options=options)
I want to download a file and I am able to do it with the code below. When I pass options to the driver, download does not start.
from selenium import webdriver
url = "http://wwwapps.tc.gc.ca/Saf-Sec-Sur/2/CCARCS-RIACC/DDZip.aspx"
driver.get(url)
driver.find_element_by_id("btnDownload").click()
I tried to pass following options, but download does not start:
from selenium import webdriver
url = "http://wwwapps.tc.gc.ca/Saf-Sec-Sur/2/CCARCS-RIACC/DDZip.aspx"
options = webdriver.ChromeOptions()
options.add_argument("download.default_directory=H:/")
options.add_argument('headless')
options.add_argument('window-size=1920x1080')
options.add_argument("disable-gpu")
with webdriver.Chrome(chrome_options=options) as driver:
driver.get(url)
driver.find_element_by_id("btnDownload").click()
I also tried:
from selenium import webdriver
url = "http://wwwapps.tc.gc.ca/Saf-Sec-Sur/2/CCARCS-RIACC/DDZip.aspx"
with webdriver.Chrome() as driver:
prefs = {
"download.default_directory": down_dir,
"download.prompt_for_download": False,
"download.directory_upgrade": True
}
options.add_experimental_option('prefs', prefs)
driver.get(url)
driver.find_element_by_id("btnDownload").click()
I would like to download the file with hidden browser window. Also, is there a way to close it just after successful download (using driver.quit())?
EDIT:
I removed duplicated driver instances - error that I made during copying pieces of code.
You instantiate your webdriver twice, delete/comment out the other line:
with webdriver.Chrome(options=options) as driver:
# driver = webdriver.Chrome(options=options)
Also not sure for Options class, I believe you should import it with:
from selenium.webdriver.chrome.options import Options
EDIT: yep, Chrome's not going to download in headless mode: SO answer.
So, the solution for you is:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
url = "http://wwwapps.tc.gc.ca/Saf-Sec-Sur/2/CCARCS-RIACC/DDZip.aspx"
def enable_download_in_headless_chrome(driver, download_dir):
# add missing support for chrome "send_command" to selenium webdriver
driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')
params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': download_dir}}
driver.execute("send_command", params)
options = Options()
options.add_argument('--headless')
driver = webdriver.Chrome(options=options)
enable_download_in_headless_chrome(driver, "H:/")
driver.get(url)
driver.find_element_by_id("btnDownload").click()
I am trying to disable all cookies when starting up either the Chrome or Firefox browser. I have seen the examples on here but they're all in Java, and some of the Selenium code is different than it is for Python.
ChromeOptions options = new ChromeOptions();
Map prefs = new HashMap();
prefs.put("profile.default_content_settings.cookies", 2);
options.setExperimentalOptions("prefs", prefs);
driver = new ChromeDriver(options);
I want to do the above, just in Python.
For Firefox:
from selenium import webdriver
fp = webdriver.FirefoxProfile()
fp.set_preference("network.cookie.cookieBehavior", 2)
browser = webdriver.Firefox(firefox_profile=fp)
Source: the FAQ, a JS selenium cookie question, and the description of Network.cookie.cookieBehavior.
For Chrome after version 45, you would need to do this (#alecxe was right up til Chrome 45 I think):
selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("prefs", {"profile.default_content_setting_values.cookies": 2})
driver = webdriver.Chrome(chrome_options=chrome_options)
The only meaningful change there is default_content_settings becomes default_content_setting_values.
It would be:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("prefs", {"profile.default_content_settings.cookies": 2})
driver = webdriver.Chrome(chrome_options=chrome_options)
tested - worked for me (Chrome 45, selenium 2.47).
You only need to change there is {"profile.default_content_setting_values.cookies": 2} becomes {"profile.block_third_party_cookies": True}.
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_experimental_option("prefs", {"profile.block_third_party_cookies": True})
driver = webdriver.Chrome(chrome_options=chrome_options)