We Keep Coding

Selenium script to download file on headless mode

This is not a duplicate. I want to achieve this on a headless linux server. No GUI, no browser profile file. I need to set profile preferences in code.
My goal: download an image file generated by a .php + .js script using python selenium firefox on full headless mode.
My issue:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.common.by import By
profile = FirefoxProfile()
profile.set_preference("browser.download.folderList", 2)
profile.set_preference("browser.download.manager.showWhenStarting", False)
profile.set_preference("browser.download.dir", "/home/ubuntu")
profile.set_preference("browser.helperApps.neverAsk.saveToDisk", "image/png")
options = Options()
options.headless = True
options.profile = profile
driver = webdriver.Firefox(options=options)
driver.get("https://old.ferramentas.eucreio.org/citacao/")
print(driver.title)
driver.close()
returns:
get_assets.py:6: DeprecationWarning: firefox_profile has been deprecated, please use an Options object
profile = FirefoxProfile()
get_assets.py:14: DeprecationWarning: Setting a profile has been deprecated. Please use the set_preference and install_addons methods
options.profile = profile
Traceback (most recent call last):
File "get_assets.py", line 16, in <module>
driver = webdriver.Firefox(options=options)
File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/firefox/webdriver.py", line 179, in __init__
RemoteWebDriver.__init__(
File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 268, in __init__
self.start_session(capabilities, browser_profile)
File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 359, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute
self.error_handler.check_response(response)
File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: Connection refused (os error 111)
If I sneak a options.set_preference() (instead of using the FirefoxProfile instance) in there it doesn't work either. Also trying to set firefox_profile as webdriver.Firefox() parameter has no different outcome.. just says everything is deprecated.
I'm having such a headache with this.. just feel something is so off here and I can't say what.
I've also exhausted the entire internet looking for a code that shed some light on this and.. just nothing.
The internet page is set to download the image as soon as it loads. And all I want is a headless download 😁
Someone with some more experience could help me please? 🙏

Firefox fails to open url when using webdriver python API

I am new to python and trying to build a selenium code to open a website via firefox.
I am using a debian stretch machine for the tests. The versions of the tools are given below :
geckodriver 0.24.0 ( 2019-01-28) , Python 2.7.13 , Mozilla Firefox 52.7.3 , selenium (3.141.0)
I see that the firefox window opens( even though in headless). But it doesnt proceed further with opening the website. The firefox instance waits for sometime after opening and then stops, the script gives error like
WebHandle.open_application("Firefox" , "http://www.google.com")
File "/home/yyyyy/yyyyy/yyyyy/yyyyy/script.py", line 49, in open_application
driver = webdriver.Firefox(firefox_options = options, executable_path='/usr/local/bin/geckodriver')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 174, in init
keep_alive=True)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in init
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused
Code :
from selenium import webdriver
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options as FirefoxOptions
options = FirefoxOptions()
options.add_argument("--headless")
print (" * Opening firefox session")
driver = webdriver.Firefox(firefox_options = options, executable_path='/usr/local/bin/geckodriver')
driver.get("https://www.google.com")
driver.maximize_window()
print(driver.title)
While looking into the geckodriver.log , I see the following errors
1624193615192 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "--headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile.Wwi0327B9dAL"
1624193616996 Marionette INFO Listening on port 2828
[Child 28357] ###!!! ABORT: Aborting on channel error.: file /build/firefox-esr-52.7.3esr/ipc/glue/MessageChannel.cpp, line 2152
[Child 28357] ###!!! ABORT: Aborting on channel error.: file /build/firefox-esr-52.7.3esr/ipc/glue/MessageChannel.cpp, line 2152
###!!! [Child][MessageChannel] Error: (msgtype=0x3E0003,name=PCompositable::Msg_Destroy) Channel error: cannot send/recv
###!!! [Child][MessageChannel] Error: (msgtype=0x3E0003,name=PCompositable::Msg_Destroy) Channel error: cannot send/recv
Thanks already for any help in fixing the issue

Try placing the geckodriver.exe file in a separate folder & drive and updating the path in executable_path.
eg:
driver = webdriver.Firefox(firefox_options = options, executable_path=r"D:/Python/drivers/geckodriver.exe")

Why does this python code work on my XUbuntu (Ubuntu 20.04) machine but not my Ubuntu 18.04 Server

So I have written this python code which is supposed to update my advertisment on a german roomshare website automatically:
from time import sleep
from selenium import webdriver
import sched, time
from selenium.webdriver import FirefoxOptions
import datetime
s = sched.scheduler(time.time, time.sleep)
def click_button_by_id(button_s, browser):
button = browser.find_element_by_id(button_s)
button.click()
def refresh_ads(sc):
opts = webdriver.ChromeOptions()
opts.add_argument("--headless")
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
browser = webdriver.Chrome(options = opts)
browser.get('https://www.wg-gesucht.de')
try:
click_button_by_id("cmpbntyestxt", browser)
except:
pass
browser.implicitly_wait(5)
login_link = browser.find_element_by_xpath('//*[#id="headbar_wrapper"]/div[2]/a[2]')
login_link.click()
sleep(2)
username_input = browser.find_element_by_id("login_email_username")
password_input = browser.find_element_by_id("login_password")
username_input.send_keys("MY_USERNAME")
password_input.send_keys("MY_PASSWORD")
login_button = browser.find_element_by_id("login_submit")
login_button.click()
sleep(2)
browser.get('https://www.wg-gesucht.de/angebot-bearbeiten.html?action=update_offer&offer_id=xxxxxxxx')
click_button_by_id("set_today_date",browser)
click_button_by_id("update_offer",browser)
sleep(2)
browser.get('https://www.wg-gesucht.de/angebot-bearbeiten.html?action=update_offer&offer_id=xxxxxxxx')
click_button_by_id("set_today_date",browser)
click_button_by_id("update_offer",browser)
print("Refreshed adds at:",datetime.datetime.now().strftime("%c"))
sleep(2)
browser.close()
s.enter(1800, 1, refresh_ads, (sc,))
s.enter(1, 1, refresh_ads, (s,))
s.run()
It works completely fine on my homemachine using this:
Ubuntu 20.04
Python 3.8.5
Chromium 88.0.4324.96
Chromedriver 1:85.0.4183.83-0ubuntu0.20.04.2
While crashing on my Server using this:
Ubuntu 18.04
Python 3.8.7 (was 3.6.9 before. I updated)
Chromium 87.0.4280.66
Chromedriver 87.0.4280.66-0ubuntu0.18.04.1
and this error message:
File "wg_gesucht_bot.py", line 66, in <module>
s.run()
File "/usr/lib/python3.6/sched.py", line 154, in run
action(*argument, **kwargs)
File "wg_gesucht_bot.py", line 23, in refresh_ads
browser = webdriver.Chrome(options = opts)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from tab crashed
(Session info: headless chrome=87.0.4280.66)
I have already tried using Firefox, it was actually what I used in the first place, and is also the reason why I still import FirefoxOptions. Because otherwise it won't work properly (it starts but doesnt get far on the webpage itself) but that is a problem I can solve afterwards.
There was this: https://stackoverflow.com/a/40379664/10811865 but updating to a 5 year old version didn't make a whole lot of sense to me
And this: https://stackoverflow.com/a/59191325/10811865
Which I also tried.
Any help is appreciated

So I found a solution for the problem:
First of all I had to update python to 3.8.5 which I did using this :
https://linuxize.com/post/how-to-install-python-3-8-on-ubuntu-18-04/
Afterwards I would encounter a no module named selenium found problem so I followed this:
https://shashanksrivastava.medium.com/how-to-fix-no-module-named-selenium-error-in-python-3-da3fd7b61485
to install in manually
Then I still encountered an error described here:
WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser
Those fixes I had already implemented so they weren't a whole lot of help but killing all chromium and chromedriver processes on my machine fixed the problem for me.
And long last I had to add this line of code:
opts.add_argument('window-size=1920x1080');
so that my code would run properly even in headless mode.

Selenium WebDriver Exception: Message: unknown error: Failed to create Chrome process

I have tried all fixes but for some reason, exceptions are coming in my code. Please help me out. The code block is trying to automate a mass WhatsApp messaging bot.
The code was adapted from a publicly available GitHub repository.
The chrome version is updated to the newest release. I am using python 3.9 environment with the latest pip installer, updated selenium library, and the most recent chromedriver extension.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from time import sleep
from urllib.parse import quote
options = Options()
#options.add_argument("user-data-dir=/tmp/tarun")
#options.add_argument("user-data-dir=C:\\Users\\anirudh_bagri\\AppData\\Local\\Google\\Chrome\\User Data")
f = open("message.txt", "r")
message = f.read()
f.close()
print('This is your message:')
print(message)
message = quote(message)
numbers = []
f = open("numbers.txt", "r")
for line in f.read().splitlines():
if line != "":
numbers.append(line)
f.close()
print('\nWe found ' + str(len(numbers)) + ' numbers in the file')
delay = 30
print('Once your browser opens up, make sure you sign in to web whatsapp and then press enter')
driver = webdriver.Chrome(executable_path=r'C:\Users\LEGION\Desktop\whatsapp-bulk-messenger-master\chromedriver.exe', options=options)
driver.get('https://web.whatsapp.com')
input()
for number in numbers:
if number == "":
continue
print('Sending message to: ' + number)
try:
url = 'https://web.whatsapp.com/send?phone=' + number + '&text=' + message
driver.get(url)
click_btn = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.CLASS_NAME , '_3M-N-')))
click_btn.click()
sleep(1)
print('Message sent to: ' + number)
except Exception:
print('Failed to send message to ' + number)
This is the exception that is coming up.
= RESTART: C:\Users\LEGION\Desktop\whatsapp-bulk-messenger-master\automator.py =
This is your message:
Hello World,
This is my text to you from automated messaging system.
Thank You
We found 1 numbers in the file
Once your browser opens up, make sure you sign in to web whatsapp and then press enter
Traceback (most recent call last):
File "C:\Users\LEGION\Desktop\whatsapp-bulk-messenger-master\automator.py", line 31, in <module>
driver = webdriver.Chrome(executable_path=r'C:\Users\LEGION\Desktop\whatsapp-bulk-messenger-master\chromedriver.exe', options=options)
File "C:\Users\LEGION\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 76, in __init__
RemoteWebDriver.__init__(
File "C:\Users\LEGION\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "C:\Users\LEGION\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Users\LEGION\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\LEGION\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Failed to create Chrome process.
>>>

This most probably happens when you run Chrome as an administrator. Try running it "not as an admin" and check once.
Just got to Chrome's properties->compatibility and change that

Python3 Selenium Webdriver times out during initialization

The following script(s):
from selenium import webdriver
wd = webdriver.Firefox()
or
from selenium import webdriver
wd = webdriver.Chrome()
will very nearly always fail on my Windows 7 machine, producing the following stack trace(s):
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\firefox\webdriver.py", line 152, in __init__
keep_alive=True)
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 98, in __init__
self.start_session(desired_capabilities, browser_profile)
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 188, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 254, in execute
response = self.command_executor.execute(driver_command, params)
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 464, in execute
return self._request(command_info[0], url, body=data)
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 488, in _request
resp = self._conn.getresponse()
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 1334, in getresponse
response.begin()
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 300, in begin
version, status, reason = self._read_status()
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 260, in _read_status
line_tmp = self.fp.readline(_MAXLINE + 1)
File "C:\Users\username\AppData\Local\Programs\Python\Python36-32\lib\socket.py", line 586, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
And the same for Chrome (except ...\webdriver\chrome\webdriver.py, obviously). This behavior is very consistent, although on rare occasions reattempting to open a webdriver within the same Python session will allow Selenium to successfully open the browser, and (so far as I've been able to tell) operate correctly from there on out.
This behavior is not observed on my Linux machine, which is on the same network; any attempt to open a webdriver works correctly.
I'm stumped, all socket timeout issues I've found through Google have been related to attempting to access a webpage, not just creating a new webdriver object. I can provide any additional information required to solve this.

Here is the Answer to your Question -
While you work with Selenium 3.4.3, chromedriver v2.30, geckodriver v0.17.0, Google Chrome 59.0 and Mozilla Firefox 53.0 through Python 3.6.1 you can consider the following options:
Chrome :
To initialize chromedriver you can consider mentioning the absolute path of the chromedriver through executable_path argument and additionally specify/use the chrome_options argument to configure the required browser properties as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path="C:\\Utility\\BrowserDrivers\\chromedriver.exe")
driver.get("https://www.google.co.in")
Firefox :
To initialize geckodriver you can consider mentioning the absolute path of the geckodriver through executable_path argument and additionally specify/use the firefox_binary argument to configure the required browser as follows:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
binary = FirefoxBinary('C:\\Program Files\\Mozilla Firefox\\firefox.exe')
caps = DesiredCapabilities().FIREFOX
driver = webdriver.Firefox(firefox_binary=binary, executable_path="C:\\Utility\\BrowserDrivers\\geckodriver.exe")
driver.get('https://stackoverflow.com')
Let me know if this Answers your Question.