We Keep Coding

Updating Selenium Python to v4.8.0 "options.headless = True" shows DeprecationWarning

I have followed the change exact but it doesn't work, are there any other bugs?
https://www.selenium.dev/blog/2023/headless-is-going-away/
import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
options = webdriver.ChromeOptions()
options.headless = True
options.page_load_strategy = 'none'
chrome_path = 'chromedriver.exe'
chrome_service = Service(chrome_path)
driver = Chrome(options=options, service=chrome_service)
driver.implicitly_wait(5)
url= "https://hk.centanet.com/findproperty/list/transaction/%E6%84%89%E6%99%AF%E6%96%B0%E5%9F%8E_3- DMHSZHHRHD?q=TiDxvVGMUUeutVzA0g1JlQ"
driver.get(url)
time.sleep(10)
contents = driver.find_element(By.CSS_SELECTOR,"div[class*='bx--structured-list-tbody']")
properties = contents.find_elements(By.CSS_SELECTOR,"div[class*='bx--structured-list-row']")
def extract_data(element):
columns = element.find_elements(By.CSS_SELECTOR,"div[class*='bx--structured-list-td']")
Date = columns[0].text
Dev = columns[1].text
Price = columns[3].text
RiseBox = columns[4].text
Area = columns[5].text
return{
"Date": Date,
"Development": Dev,
"Consideration": Price,
"Change": RiseBox,
"Area": Area
}
data = []
for property in properties:
extracted_data = extract_data(property)
data.append(extracted_data)
df = pd.DataFrame(data)
df.to_csv("result.csv", index=False)
Comment was as follows:
Selenium.py:10: DeprecationWarning: headless property is deprecated, instead use add_argument('--headless') or add_argument('--headless=new')
options.headless = True
I tried the changes suggested by https://www.selenium.dev/blog/2023/headless-is-going-away/
but it doesn't work

This warning message...
DeprecationWarning: headless property is deprecated, instead use add_argument('--headless') or add_argument('--headless=new') options.headless = True
...is inline with the Selenium Blog post with the heading Headless is Going Away!
Details
Chromium team have released the Native Headless mode which is now officially the new Headless mode. This functionality is available with:
Chromium v109.0.5400.0
ChromeDriver v109.0.5414.25
Selenium v4.8.0
The new syntax requires --headless=new to be passed as an argument as follows:
Java:
ChromeOptions options = new ChromeOptions();
options.addArguments("--headless=new");
WebDriver driver = new ChromeDriver(options);
driver.get("https://selenium.dev);
driver.quit();
Python:
options = ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
driver.get('http://selenium.dev')
driver.quit()
Javascript:
let driver = await env
.builder()
.setChromeOptions(options.addArguments('--headless=new'))
.build();
await driver.get('https://selenium.dev');
await driver.quit();
CSharp:
var options = new ChromeOptions();
options.AddArgument("--headless=new");
var driver = new ChromeDriver(options);
driver.Navigate().GoToUrl("https://selenium.dev");
driver.Quit();
Ruby:
options = Selenium::WebDriver::Options.chrome(args: ['--headless=new'])
driver = Selenium::WebDriver.for :chrome, options: options
driver.get('https://selenium.dev')
driver.quit

Try this configuration:
# pip install selenium==4.8.0
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
options=Options()
options.add_argument("--headless=new")
options.add_experimental_option("excludeSwitches", ["enable-logging"]) #Optional
service=Service(f"path\chromebrowser.exe")
browser=webdriver.Chrome(service=service, options=options)
print(browser.name)
pythonseleniumwebdriveroptions

Python using selenium webdriver to fill a form, when submitting the form, chrome is saying no internet connection is available

I am using Selenium webdriver to fill out a form, the fields are good, but when I do a .submit using the login button, or do a .submit after the password field, the chrome browser will say "Unable to connect to internet. Please check your internet connection."
the internet is definitely up since I am able to open other websites on another browser (including a new session of chrome).
This is my code:
import time
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = webdriver.ChromeOptions();
chrome_options.add_experimental_option("excludeSwitches", ['enable-automation']);
options = Options()
# options.add_argument('start-maximumized')
# options.add_argument('disable-infobars')
chrome_options.add_argument('no-sandbox')
PATH = "c:\scripts\chromedriver.exe"
# driver = webdriver.Chrome(executable_path=PATH)
driver = webdriver.Chrome(chrome_options=chrome_options,
executable_path='c:\scripts\chromedriver.exe')
driver.get('https://stupidwebsite.org')
time.sleep(1)
input_username = driver.find_element_by_id('username')
time.sleep(1)
input_username.send_keys("myself#email.com")
input_password = driver.find_element_by_id('password')
input_password.send_keys("mypassword#123")
input_password.submit()
# esubmit=driver.find_element_by_xpath
('/html/body/div/div[1]/div[2]/div/div/form/div[2]/div/button')
# esubmit = driver.find_element_by_name('MuiButton-label')
# esubmit.submit()
time.sleep(2)
# driver.quit()
I changed it a bit, and tried out google, and it works:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
#set chromodriver.exe path
driver = webdriver.Chrome(executable_path="C:\scripts\chromedriver.exe")
driver.implicitly_wait(0.5)
#launch URL
driver.get("https://www.google.com/")
#identify search box
m = driver.find_element_by_name("q")
#enter search text
m.send_keys("Tutorialspoint")
time.sleep(0.2)
#perform Google search with Keys.ENTER
m.send_keys(Keys.ENTER)
Please help me out. Thank you.

How to use Firefox profiles in Selenium Python (windows)

I'm unable lo load my Firefox profile on the Selenium Webdriver.
i have tried several ways commented here on SO, but non did the trick. Either i loose connection to the driver or the profile does not get loaded.
this is what i have so far:
from selenium import webdriver
options = Options()
options.headless = False
fp = webdriver.FirefoxProfile(r'C:\\Users\\xxXX\\AppData\\Local\\Mozilla\\Firefox\\Profiles\\y73h6ogb.default-release')
driver = webdriver.Firefox(fp, options=options, executable_path='geckodriver/geckodriver.exe')
driver.get(URL)

According to the documentation you can load a profile like this:
from selenium.webdriver import Firefox, FirefoxOptions, FirefoxProfile
options = FirefoxOptions()
options.headless = True
profile = FirefoxProfile("PATH_TO_PROFILE\\1234asdf.some_profile")
driver = Firefox(
firefox_profile=profile,
executable_path="PATH_TO_DRIVER\\geckodriver.exe",
options=options,
)
driver.get("https://python.org")
driver.close()
Or by adding the profile to the options:
...
options = FirefoxOptions()
options.headless = True
options.profile = FirefoxProfile("PATH_TO_PROFILE\\1234asdf.some_profile")
driver = Firefox(
executable_path="PATH_TO_DRIVER\\geckodriver.exe",
options=options,
)
...

By using webdriver for firefox, You will be able to do that!

Selenium raises BadStatusLine error

Hi I'm using selenium in my web app on heroku. I used it to login to instagram but it raises a BadStatusLine error whenever selenium is sending keys to the user field/password field.
My code:
display = Display(visible=0, size=(800, 600))
display.start()
options = Options()
options.binary_location = "/app/.apt/usr/bin/google-chrome-stable"
driver = webdriver.Chrome(chrome_options = options)
driver.get("http://www.instagram.com/")
#click login button to show login form
driver.implicitly_wait(20) # seconds
login_btn = driver.find_element_by_link_text("Log in")
login_btn.click()
#enter username and password
driver.implicitly_wait(20) # seconds
driver.find_element_by_name("username").send_keys(login)
driver.find_element_by_name("password").send_keys(password)
Error:

For anyone who need, you will need
-Two buildpacks
-Set stack to cedar-14
-Have chromedriver in your files. Point to it and set chromedriver options
from selenium.webdriver.chrome.options import Options
dir_path = os.path.dirname(os.path.realpath(__file__))
chromedriver_path = os.path.join(dir_path, "chromedriver")
options = Options()
options.binary_location = "/app/.apt/usr/bin/google-chrome-stable"
driver = webdriver.Chrome(executable_path = chromedriver_path, chrome_options = options)

Using selenium: How to keep logged in after closing Driver in Python

I want to get my Whatsapp web (web.whatsapp.com) logged in, at the second time opening the Whatsapp web on chrome driver. Following is my code based on Python need your help.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_path = r"chromedriver.exe"
options = Options();
options.add_argument("user-data-
dir=C:/Users/Username/AppData/Local/Google/Chrome/User Data");
#options.add_argument("--start-maximized");
driver = webdriver.Chrome(chrome_path,chrome_options=options);
#driver = webdriver.Chrome();
driver.get('https://web.whatsapp.com/')

I tried on my Mac, below code and it worked perfectly fine, I don't need to login again
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("user-data-dir=/tmp/tarun")
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://web.whatsapp.com/')
driver.quit()
For window you can try changing the path as below
options.add_argument("user-data-dir=C:\\Users\\Username\\AppData\\Local\\Google\\Chrome\\User Data")

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("user-data-dir=C:\\Users\\Username\\AppData\\Local\\Google\\Chrome\\User Data")
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://web.whatsapp.com/')
driver.quit()
Here it is for Windows. Works perfect on Python 3.6