We Keep Coding

Python Selenium and Chromedriver CentOS8

I have a python script which use selenium and chromedriver.
It runs on my CentOS8 VPS perfectly for 3 days without any problem.
But since this morning, the script launched, wait almost 80 secondes and display this :
[12/Jan/2021 23:04:51] ERROR - Failed : Message: chrome not reachable
Traceback (most recent call last):
File "script.py", line 55, in <module>
driver = launch()
File "script.py", line 37, in launch
browser = webdriver.Chrome('/usr/bin/chromedriver',chrome_options=chrome_options)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: chrome not reachable
No modification have been made, why does it fail now ?
I don't have any screen on my VPS so I can't see more information.
Here is some info :
yum info on chromedriver :
Nom : chromedriver
Version : 87.0.4280.88
Publication : 1.el8
Architecture : x86_64
Taille : 27 M
Source : chromium-87.0.4280.88-1.el8.src.rpm
Dépôt : #System
Depuis le dé : epel
google-chrome --version :
Google Chrome 87.0.4280.141
Begin of the script :
from dotenv import load_dotenv
from logger import logger as l
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.chrome.options import Options
import time
import sys
import subprocess
load_dotenv(verbose=True)
dotenv_path = '.env'
load_dotenv(dotenv_path)
def launch():
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
browser = webdriver.Chrome('/usr/bin/chromedriver',chrome_options=chrome_options)
l.info('Started Chrome')
return browser

Problem solved but don't understand how.
I just restart my VPS (reboot), and ... it works again.
Weird
EDIT : Find why !
I just made a mistake at the end of my script :
b.close();
But "b" don't exist, my driver variable name is "driver".
The exception was catched and not displayed, so I don't saw anything.
But today, I launch a "top" command, and see all "chrome" process running in background.
Probably after several days, memory was full, and Chrome can't launch.
The error was not clear but anyway, it was my fault.

Thumb rule
A common cause for Chrome to crash during startup is running Chrome as root user (administrator) on Linux. While it is possible to work around this issue by passing --no-sandbox flag when creating your WebDriver session, such a configuration is unsupported and highly discouraged. You need to configure your environment to run Chrome as a regular user instead.
Solution
Remove the following chrome_options:
--no-sandbox
and execute your code as a non root user.
Outro
Here is the link to the Sandbox story.

Selenium not using Default Chrome Profile

Selenium will not load my default Chrome Profile and I cannot figure out why. I have tried both Profile 1 and a Default profile with the same error (below). I have confirmed with Task Manager that all Chrome windows are shut down before running this code. Any thoughts?
from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd
import numpy as np
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
import os
os.system("taskkill /f /im geckodriver.exe /T")
os.system("taskkill /f /im chromedriver.exe /T")
os.system("taskkill /f /im IEDriverServer.exe /T")
os.system("taskkill /f /im chrome.exe /T")
driver2 = r"C:\Users\xxx\.wdm\drivers\chromedriver\87.0.4280.20\win32\chromedriver.exe"
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Users\xxx\\AppData\\Local\\Google\\Chrome\\User Data\\Profile 1")
driver = webdriver.Chrome(executable_path=driver2, chrome_options=options)
driver.get("https://www.google.co.in")
Traceback (most recent call last):
File "C:\Users\xxx\OneDrive\Python\pyReportRun.py", line 16, in <module>
driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)
File "C:\Python38\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 76, in __init__
RemoteWebDriver.__init__(
File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Python38\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Could not remove old devtools port file. Perhaps the given user-data-dir at C:\Users\xxx\AppData\Local\Google\Chrome\User Data\Profile 1 is still attached to a running Chrome or Chromium process

chrom_options.add_argument("user-data-dir=C:\\Users\robert.car\\AppData\\Local\\Google\\Chrome\\User Data")
chrom_options.add_argument("profile-directory=Profile 1")
user-data-dir considers profile as default , and you don't have to specify that . If its something else specify it through profile-directory argument
Step to create a profile:
open : chrome://version in address bar
copy the user dir folder completely to eg c:\tmp\newdir
open the copied user data (newdir) and search for folder called Default . This is the profile folder.
rename the Default folder as "Profile 1"
Now to use this :
chrom_options.add_argument("user-data-dir=c:\\tmp\\newdir")
chrom_options.add_argument("profile-directory=Profile 1")

This error message...
selenium.common.exceptions.WebDriverException: Message: unknown error: Could not remove old devtools port file. Perhaps the given user-data-dir at C:\Users\xxx\AppData\Local\Google\Chrome\User Data\Profile 1 is still attached to a running Chrome or Chromium process
...implies that the ChromeDriver was unable to initiate/spawn a new Browsing Context i.e. Chrome Browser session as the user-data-dir which you have passed an an argument is still attached to a running Chrome Browsing Context.
Details
Using Default Chrome Profile for Test Automation will be against all the best practices as the Default Chrome Profile may contain either/all of the following:
browser settings
Extensions
Bookmarks
Apps
Saved Passwords
Browsing History
etc
So the Default Chrome Profile may not be in compliance with you Test Specification and may occasionally raise exceptions while trying to load. Hence you should always use a customized Chrome Profile.
You can find a detailed discussion in How to open a Chrome Profile through --user-data-dir argument of Selenium
If your usecase still warrants to use the Default Chrome Profile you need to ensure that all the google-chrome, chromium or selenium-chromedriver are stopped/killed and you can follow the below mentioned details.
Here you can find a detailed discussion on Selenium : How to stop geckodriver process impacting PC memory, without calling driver.quit()?
Location of Default Chrome Profile
As per the documentation in How to Find Your Chrome Profile Folder on Windows, Mac, and Linux the location for Chrome’s default profile folder differs depending on your platform. The locations are:
Windows 7, 8.1, and 10: C:\Users\<username>\AppData\Local\Google\Chrome\User Data\Default
Mac OS X El Capitan: Users/<username>/Library/Application Support/Google/Chrome/Default
Linux: /home/<username>/.config/google-chrome/default
You need to replace <username> with the name of your user folder. The default profile folder is simply named Default (or default in Linux). However, if you’ve created additional profiles, their folder names are not as obvious. The name you assigned to the profile when you created it displays on a name button on the right side of the title bar on the Chrome window. Unfortunately, the name Chrome uses on the associated profile folder is a generic, numbered name like Profile 3.
If you need to know any of the Chrome Profile's folder name, you simply need to access chrome://version in the address bar and press Enter.
Snapshot:
The Profile Path shows the location of the current profile. For example, the location of my Default profile in my Windows 10 system is C:\Users\Soma Bhattacharjee\AppData\Local\Google\Chrome\User Data\Default. You can select the path and copy it and paste it into File Explorer in Windows, the Finder on OS X or into a file manager like Nautilus in Linux to access that folder.
Sample Code (Windows 10)
Finally, to access the Default Chrome Profile you can use the following Python based solution:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Users\\username\\AppData\\Local\\Google\\Chrome\\User Data\\Default")
driver = webdriver.Chrome(executable_path=r'C:\WebDrivers\chromedriver.exe', chrome_options=options)
driver.get("https://www.google.co.in")
You can find a detailed discussion in How to use Chrome Profile in Selenium Webdriver Python 3

Unable to open site url in browser

I have created simple basic automation script in Python using Selenium..
Getting unwanted exception.
File:-
import pandas as pd
from pandas import ExcelWriter
from selenium import webdriver
import selenium as sel
# Data = pd.read_excel(r"C:\Users\Admin\PycharmProjects\Web_Automation_Form_Filling\challenge.xlsx",sheet_name="Sheet1")
# browser = webdriver.Chrome(executable_path=r'C:\Program Files (x86)\Google\Chrome\Application\chrome.exe')
browser = webdriver.Chrome("C:\Program Files (x86)\Google\Chrome\Application\chrome.exe");
browser.sleep(1000);
browser.get("http://www.python.org")
Error log:-
C:\Users\Admin\PycharmProjects\Web_Automation_Form_Filling\venv\Scripts\python.exe C:/Users/Admin/PycharmProjects/Web_Automation_Form_Filling/venv/Web_Auto_Filling.py
Traceback (most recent call last):
File "C:/Users/Admin/PycharmProjects/Web_Automation_Form_Filling/venv/Web_Auto_Filling.py", line 10, in <module>
browser = webdriver.Chrome("C:\Program Files (x86)\Google\Chrome\Application\chrome.exe");
File "C:\Users\Admin\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__
self.service.start()
File "C:\Users\Admin\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\common\service.py", line 98, in start
self.assert_process_still_running()
File "C:\Users\Admin\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\common\service.py", line 109, in assert_process_still_running
raise WebDriverException(
selenium.common.exceptions.WebDriverException: Message: Service C:\Program Files (x86)\Google\Chrome\Application\chrome.exe unexpectedly exited. Status code was: 0
Process finished with exit code 1
Any suggestion will be appreciated..
Thanks...

instead of chrome application try providing the chrome driver instead
more information on the site : https://sites.google.com/a/chromium.org/chromedriver/getting-started
Sample code :
import time
from selenium import webdriver
driver = webdriver.Chrome('/path/to/chromedriver') # Optional argument, if not specified will search path.
driver.get('http://www.google.com/');
time.sleep(5) # Let the user actually see something!
search_box = driver.find_element_by_name('q')
search_box.send_keys('ChromeDriver')
search_box.submit()
time.sleep(5) # Let the user actually see something!
driver.quit()
Download the ChromeDriver binary for your platform under the downloads section of this site
reference link to download : chrome driver

This code should work (better to use firefox for selenium):
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
# noinspection PyUnresolvedReferences
import wget
DesiredCapabilities.PHANTOMJS[
"phantomjs.page.settings.userAgent"
] = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:16.0) Gecko/20121026 Firefox/16.0"
if browser == "firefox":
driver = webdriver.Firefox()
else:
driver = webdriver.PhantomJS(
CFG_phantomjs
) # r"D:/_devs/webserver/phantomjs-1.9.8/phantomjs.exe"
driver.get("https://tourwebsite")
username = driver.find_element_by_id("login_field")
password = driver.find_element_by_id("password")
username.clear()

The problem here in your codes is that you are passing chrome executable path rather than passing the path to chromedriver which is a different executable.
An appropriate version of chromedriver can be downloaded from here according to your Chrome version.
For more info, you can refer to the chromedriver documentation here.
And your final code should be something like:
from selenium import webdriver
path = 'C:/Users/Avinash/Downloads/chromedriver.exe'
driver = webdriver.Chrome(path)
driver.get('http://www.google.com/');
#..here what ever you want to do with page here
driver.quit()

python selenium not able to open Firefox browser

I am using Python with selenium. I am getting following error when I am trying to open Firefox using python selenium
>>> driver = webdriver.Firefox()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriver\firefox\webdriver.py", line 59, in __init__
self.binary, timeout),
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriver\firefox\extension_connection.py", line 47, in __init__
self.binary.launch_browser(self.profile)
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriver\firefox\firefox_binary.py", line 61, in launch_browser
self._wait_until_connectable()
File "C:\Python27\lib\site-packages\selenium-2.39.0-py2.7.egg\selenium\webdriver\firefox\firefox_binary.py", line 105, in _wait_until_connectable
self.profile.path, self._get_firefox_output()))
selenium.common.exceptions.WebDriverException: Message: 'Can\'t load the profile. Profile Dir: c:\\users\\workshop\\appdata\\local\\temp\\tmpzcofay Firefox output: 1422716384881\taddons.manager\tDEBUG
\tLoaded provider scope for resource://gre/modules/addons/XPIProvider.jsm: ["XPIProvider"]\r\n1422716384882\taddons.manager\tDEBUG\tLoaded provider scope for resource://gre/modules/LightweightThemeMan
ager.jsm: ["LightweightThemeManager"]\r\n1422716384884\taddons.xpi\tDEBUG\tstartup\r\n1422716384886\taddons.xpi\tINFO\tMapping fxdriver#googlecode.com to c:\\users\\workshop\\appdata\\local\\temp\\tmp
zcofay\\extensions\\fxdriver#googlecode.com\r\n1422716384887\taddons.xpi\tINFO\tMapping {F17C1572-C9EC-4e5c-A542-D05CBB5C5A08} to C:\\Program Files (x86)\\DAP\\DAPFireFox\r\n1422716384887\taddons.xpi\
tINFO\tMapping {82AF8DCA-6DE9-405D-BD5E-43525BDAD38A} to C:\\Program Files (x86)\\Mozilla Firefox\\browser\\extensions\\{82AF8DCA-6DE9-405D-BD5E-43525BDAD38A}.xpi\r\n1422716384888\taddons.xpi\tINFO\tM
apping {972ce4c6-7e08-4474-a285-3208198ce6fd} to C:\\Program Files (x86)\\Mozilla Firefox\\browser\\extensions\\{972ce4c6-7e08-4474-a285-3208198ce6fd}\r\n1422716384888\taddons.xpi\tDEBUG\tSkipping una
vailable install location app-system-share\r\n1422716384888\taddons.xpi\tDEBUG\tSkipping unavailable install location app-system-local\r\n1422716384889\taddons.xpi\tINFO\tMapping {74c841e3-b59f-479e-8
d7a-e26a942a87c8} to C:\\Program Files (x86)\\WinZip Courier\\FFExt\r\n1422716384889\taddons.xpi\tINFO\tMapping {01A8CA0A-4C96-465b-A49B-65C46FAD54F9} to C:\\Program Files (x86)\\Adobe\\Adobe Contribu
te CS5.1\\Plugins\\FirefoxPlugin\\{01A8CA0A-4C96-465b-A49B-65C46FAD54F9}\r\n1422716384889\taddons.xpi\tINFO\tMapping web2pdfextension#web2pdf.adobedotcom to C:\\Program Files (x86)\\Adobe\\Acrobat 10.
0\\Acrobat\\Browser\\WCFirefoxExtn\r\n1422716384889\taddons.xpi\tINFO\tMapping daplinkchecker#speedbit.com to C:\\Program Files (x86)\\DAP\\daplinkchecker\r\n1422716384890\taddons.xpi\tINFO\tMapping s
earchpredict#speedbit.com to C:\\Program Files (x86)\\SearchPredict\\PRFireFox\r\n1422716384890\taddons.xpi\tINFO\tMapping {0329E7D6-6F54-462D-93F6-F5C3118BADF2} to C:\\Program Files (x86)\\SPEEDbit V
ideo Downloader\\SPFireFox\r\n1422716384890\taddons.xpi\tINFO\tMapping adapter#gingersoftware.com to C:\\Program Files (x86)\\Ginger\\Mozilla\\adapter#gingersoftware.com\r\n1422716384891\taddons.xpi\t
DEBUG\tcheckForChanges\r\n'
>>> driver = webdriver.Firefox()

You need to provide path:
from selenium import webdriver
ff_profile = webdriver.FirefoxProfile()
browser = webdriver.Firefox(firefox_profile=ff_profile)
Please check if it works for You (it does for me, whereas without path i get same error as You do).

Your selenium version is quite old.
You have to install the latest version 2.53, it should solve your problem.

Remember to introduce selenium and firefox in your system path then you can use this code, some times depend of the Firefox version that you have you will need install the gekodriver too
from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
driver = webdriver.Firefox()
driver.get("YOUR_URL")
If you didn't introduce them in the system path you can use this other code:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary("C:\Program Files (x86)\Mozilla Firefox\Firefox.exe")#path of Firefox.exe
driver = webdriver.Firefox(firefox_binary=binary)
driver.get("YOUR_URL")

You need to import the correct modules. Don't forget to call them. Check my example below:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get("Insert URL here")
Run that and it should work. Let me know if it doesn't.

Trouble Getting Chrome Through Selenium Hub

I am using Selenium Webdriver, 2.25 I have a local hub set up with this json setting for chrome and firefox:
[
{
"browserName": "firefox",
"maxInstances": 5,
"seleniumProtocol": "WebDriver"
},
{
"browserName": "chrome",
"maxInstances": 5,
"seleniumProtocol": "WebDriver"
}
],
I can start a webdriver firefox session like this:
capability = getattr(webdriver.DesiredCapabilities, "FIREFOX")
dd=webdriver.Remote('http://localhost:4444/wd/hub', capability)
which works fine
but if I try to start a Chrome session like this:
capability = getattr(webdriver.DesiredCapabilities, "CHROME")
dd=webdriver.Remote('http://localhost:4444/wd/hub', capability)
I get this error:
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 62, in init
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 104, in start_session
'desiredCapabilities': desired_capabilities,
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 155, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 147, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: None ; Stacktrace: Method innerGet threw an error in None
But I can start a direct connection to Chrome like this:
dd=webdriver.Chrome()
Without any problem.
What can I do to get to Chrome through my Selenium Hub?

I had EXACTLY the same problem.
The thing is, unlike Firefox, Chrome needs separate chromdriver.exe to act as bridge between browser and driver.
From the documentation:
The ChromeDriver consists of three separate pieces. There is the
browser itself ("chrome"), the language bindings provided by the
Selenium project ("the driver") and an executable downloaded from the
Chromium project which acts as a bridge between "chrome" and the
"driver". This executable is called "chromedriver", but we'll try and
refer to it as the "server" in this page to reduce confusion.
Download chromdriver.exe here
And put it in your chrome binary dir.
I then use a .bat file to launch my hub with this listing:
java -Dwebdriver.chrome.driver="C:\Program Files (x86)\Google\Chrome\Application\chromedriver.exe" -jar D:\soft\selenium-server-standalone-2.29.0.jar
I then execute the following Python code on my Linux box, it worked flawlessly once I put chromedriver.exe in the Chrome dir and launched the hub with correct path parameters:
from selenium import webdriver
url = "http://192.168.1.115:4444/wd/hub"
driver = webdriver.Remote(command_executor = url, desired_capabilities = {'browserName':'chrome'})
driver.get("http://google.com")
Hope this helps you and the others with the same problem. Finding the solution was of course not to take firefox approach for granted and RTFM:
Chrome driver documentation

You need to setup the chrome driver, info about that here
UPDATE
Based on a sample json setup file
and steps provided in the first link, seems like the browser name should not be in Upper but in fact lower case.
So change CHROME to chrome
Example
WebDriver driver = new RemoteWebDriver("http://localhost:9515", DesiredCapabilities.chrome());
driver.get("http://www.google.com");
and in your case, I would assume
dd=webdriver.Remote('http://localhost:4444/wd/hub', DesiredCapabilities.chrome())