We Keep Coding

Selenium script to download file on headless mode

This is not a duplicate. I want to achieve this on a headless linux server. No GUI, no browser profile file. I need to set profile preferences in code.
My goal: download an image file generated by a .php + .js script using python selenium firefox on full headless mode.
My issue:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.common.by import By
profile = FirefoxProfile()
profile.set_preference("browser.download.folderList", 2)
profile.set_preference("browser.download.manager.showWhenStarting", False)
profile.set_preference("browser.download.dir", "/home/ubuntu")
profile.set_preference("browser.helperApps.neverAsk.saveToDisk", "image/png")
options = Options()
options.headless = True
options.profile = profile
driver = webdriver.Firefox(options=options)
driver.get("https://old.ferramentas.eucreio.org/citacao/")
print(driver.title)
driver.close()
returns:
get_assets.py:6: DeprecationWarning: firefox_profile has been deprecated, please use an Options object
profile = FirefoxProfile()
get_assets.py:14: DeprecationWarning: Setting a profile has been deprecated. Please use the set_preference and install_addons methods
options.profile = profile
Traceback (most recent call last):
File "get_assets.py", line 16, in <module>
driver = webdriver.Firefox(options=options)
File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/firefox/webdriver.py", line 179, in __init__
RemoteWebDriver.__init__(
File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 268, in __init__
self.start_session(capabilities, browser_profile)
File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 359, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 424, in execute
self.error_handler.check_response(response)
File "/home/ubuntu/.local/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: Connection refused (os error 111)
If I sneak a options.set_preference() (instead of using the FirefoxProfile instance) in there it doesn't work either. Also trying to set firefox_profile as webdriver.Firefox() parameter has no different outcome.. just says everything is deprecated.
I'm having such a headache with this.. just feel something is so off here and I can't say what.
I've also exhausted the entire internet looking for a code that shed some light on this and.. just nothing.
The internet page is set to download the image as soon as it loads. And all I want is a headless download 😁
Someone with some more experience could help me please? 🙏

Why does this python code work on my XUbuntu (Ubuntu 20.04) machine but not my Ubuntu 18.04 Server

So I have written this python code which is supposed to update my advertisment on a german roomshare website automatically:
from time import sleep
from selenium import webdriver
import sched, time
from selenium.webdriver import FirefoxOptions
import datetime
s = sched.scheduler(time.time, time.sleep)
def click_button_by_id(button_s, browser):
button = browser.find_element_by_id(button_s)
button.click()
def refresh_ads(sc):
opts = webdriver.ChromeOptions()
opts.add_argument("--headless")
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
browser = webdriver.Chrome(options = opts)
browser.get('https://www.wg-gesucht.de')
try:
click_button_by_id("cmpbntyestxt", browser)
except:
pass
browser.implicitly_wait(5)
login_link = browser.find_element_by_xpath('//*[#id="headbar_wrapper"]/div[2]/a[2]')
login_link.click()
sleep(2)
username_input = browser.find_element_by_id("login_email_username")
password_input = browser.find_element_by_id("login_password")
username_input.send_keys("MY_USERNAME")
password_input.send_keys("MY_PASSWORD")
login_button = browser.find_element_by_id("login_submit")
login_button.click()
sleep(2)
browser.get('https://www.wg-gesucht.de/angebot-bearbeiten.html?action=update_offer&offer_id=xxxxxxxx')
click_button_by_id("set_today_date",browser)
click_button_by_id("update_offer",browser)
sleep(2)
browser.get('https://www.wg-gesucht.de/angebot-bearbeiten.html?action=update_offer&offer_id=xxxxxxxx')
click_button_by_id("set_today_date",browser)
click_button_by_id("update_offer",browser)
print("Refreshed adds at:",datetime.datetime.now().strftime("%c"))
sleep(2)
browser.close()
s.enter(1800, 1, refresh_ads, (sc,))
s.enter(1, 1, refresh_ads, (s,))
s.run()
It works completely fine on my homemachine using this:
Ubuntu 20.04
Python 3.8.5
Chromium 88.0.4324.96
Chromedriver 1:85.0.4183.83-0ubuntu0.20.04.2
While crashing on my Server using this:
Ubuntu 18.04
Python 3.8.7 (was 3.6.9 before. I updated)
Chromium 87.0.4280.66
Chromedriver 87.0.4280.66-0ubuntu0.18.04.1
and this error message:
File "wg_gesucht_bot.py", line 66, in <module>
s.run()
File "/usr/lib/python3.6/sched.py", line 154, in run
action(*argument, **kwargs)
File "wg_gesucht_bot.py", line 23, in refresh_ads
browser = webdriver.Chrome(options = opts)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from tab crashed
(Session info: headless chrome=87.0.4280.66)
I have already tried using Firefox, it was actually what I used in the first place, and is also the reason why I still import FirefoxOptions. Because otherwise it won't work properly (it starts but doesnt get far on the webpage itself) but that is a problem I can solve afterwards.
There was this: https://stackoverflow.com/a/40379664/10811865 but updating to a 5 year old version didn't make a whole lot of sense to me
And this: https://stackoverflow.com/a/59191325/10811865
Which I also tried.
Any help is appreciated

So I found a solution for the problem:
First of all I had to update python to 3.8.5 which I did using this :
https://linuxize.com/post/how-to-install-python-3-8-on-ubuntu-18-04/
Afterwards I would encounter a no module named selenium found problem so I followed this:
https://shashanksrivastava.medium.com/how-to-fix-no-module-named-selenium-error-in-python-3-da3fd7b61485
to install in manually
Then I still encountered an error described here:
WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser
Those fixes I had already implemented so they weren't a whole lot of help but killing all chromium and chromedriver processes on my machine fixed the problem for me.
And long last I had to add this line of code:
opts.add_argument('window-size=1920x1080');
so that my code would run properly even in headless mode.

Selenium WebDriver Exception: Message: unknown error: Failed to create Chrome process

I have tried all fixes but for some reason, exceptions are coming in my code. Please help me out. The code block is trying to automate a mass WhatsApp messaging bot.
The code was adapted from a publicly available GitHub repository.
The chrome version is updated to the newest release. I am using python 3.9 environment with the latest pip installer, updated selenium library, and the most recent chromedriver extension.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from time import sleep
from urllib.parse import quote
options = Options()
#options.add_argument("user-data-dir=/tmp/tarun")
#options.add_argument("user-data-dir=C:\\Users\\anirudh_bagri\\AppData\\Local\\Google\\Chrome\\User Data")
f = open("message.txt", "r")
message = f.read()
f.close()
print('This is your message:')
print(message)
message = quote(message)
numbers = []
f = open("numbers.txt", "r")
for line in f.read().splitlines():
if line != "":
numbers.append(line)
f.close()
print('\nWe found ' + str(len(numbers)) + ' numbers in the file')
delay = 30
print('Once your browser opens up, make sure you sign in to web whatsapp and then press enter')
driver = webdriver.Chrome(executable_path=r'C:\Users\LEGION\Desktop\whatsapp-bulk-messenger-master\chromedriver.exe', options=options)
driver.get('https://web.whatsapp.com')
input()
for number in numbers:
if number == "":
continue
print('Sending message to: ' + number)
try:
url = 'https://web.whatsapp.com/send?phone=' + number + '&text=' + message
driver.get(url)
click_btn = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.CLASS_NAME , '_3M-N-')))
click_btn.click()
sleep(1)
print('Message sent to: ' + number)
except Exception:
print('Failed to send message to ' + number)
This is the exception that is coming up.
= RESTART: C:\Users\LEGION\Desktop\whatsapp-bulk-messenger-master\automator.py =
This is your message:
Hello World,
This is my text to you from automated messaging system.
Thank You
We found 1 numbers in the file
Once your browser opens up, make sure you sign in to web whatsapp and then press enter
Traceback (most recent call last):
File "C:\Users\LEGION\Desktop\whatsapp-bulk-messenger-master\automator.py", line 31, in <module>
driver = webdriver.Chrome(executable_path=r'C:\Users\LEGION\Desktop\whatsapp-bulk-messenger-master\chromedriver.exe', options=options)
File "C:\Users\LEGION\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 76, in __init__
RemoteWebDriver.__init__(
File "C:\Users\LEGION\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "C:\Users\LEGION\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Users\LEGION\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\LEGION\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Failed to create Chrome process.
>>>

This most probably happens when you run Chrome as an administrator. Try running it "not as an admin" and check once.
Just got to Chrome's properties->compatibility and change that

Error on Connect Selenium Driver to an Existing Chrome Browser Instance [duplicate]

This question already has answers here:
How can I reconnect to the browser opened by webdriver with selenium?
(4 answers)
Closed 2 years ago.
I was following this tutorial (link below), but an error is happening that I don't know how I can solve it.
https://medium.com/#harith.sankalpa/connect-selenium-driver-to-an-existing-chrome-browser-instance-41435b67affd
I am trying to use the browser that is open to perform a search, because it is already logged into the account that I need.
I'm using chrome --remote-debugging-port=1024 to open chrome, after I execute this code below.
I found some solutions, but none solved my problem, either because the solution was in Java and I didn't understand or I didn't know how to rewrite in python.
Code:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options as ChromeOptions
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
if __name__ == '__main__':
options = ChromeOptions()
options.add_argument('start-maximized')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--remote-debugging-port=1024')
options.add_argument('--disable-setuid-sandbox')
options.add_experimental_option("debuggerAddress", "localhost:1024")
options.binary_location = "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe"
d = DesiredCapabilities.CHROME
d["loggingPrefs"] = {"browser": "ALL"}
driver = webdriver.Chrome(
executable_path=ChromeDriverManager().install(),
options=options,
desired_capabilities=d
)
driver.get("https://google.com.br")
Error:
Traceback (most recent call last):
File "C:/Users/danit/Desktop/project/main.py", line 22, in <module>
desired_capabilities=d
File "C:\Users\danit\Desktop\project\venv\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "C:\Users\danit\Desktop\project\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "C:\Users\danit\Desktop\project\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Users\danit\Desktop\project\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\danit\Desktop\project\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at localhost:1024
from chrome not reachable
Thanks a lot.

The error says chrome is not reachable. Most probably that instance has been deleted
I am showing a simple method to save session data( all cookies will be saved ) and then load selenium to load from that instance.
Look at the following example
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
session = "mySession"
chrome_driver_path = '/home/aahnik/Downloads/apps/chromedriver'
whatsapp_web_url = "https://web.whatsapp.com/"
chrome_options = Options()
chrome_options.add_argument(f'--user-data-dir={session}')
driver = webdriver.Chrome(
options=chrome_options, executable_path=chrome_driver_path)
driver.get(whatsapp_web_url)
Now execute this code. WhatsApp web will open. Login by scanning the QR Code.
Now close the window, and then terminate the program.
Now will see a folder named mySession in your current user directory.
Execute this code again.
This time you will find that you are already logged into WhatsApp.
Hope this helped.

unknown error: failed to wait for extension background page to load: chrome-extension error loading an extension to Chrome Headless using Selenium

I try to run chromedriver via selenium in headless mode.
IMPORTANT
The code runs perfectly fine if I eliminate the following code lines (but is not headless):
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
This is the error I get when I try to implement the headless argument:
Traceback (most recent call last):
File "camel.py", line 83, in <module>
executable_path=executable_path)
File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to wait for extension background page to load: chrome-extension://jkompbllimaoekaogchhkmkdogpkhojg/_generated_background_page.html
from unknown error: page could not be found: chrome-extension://jkompbllimaoekaogchhkmkdogpkhojg/_generated_background_page.html
This are the lines 81, 82 and 83
chrome_options.add_extension(extension_path)
driver = webdriver.Chrome(options=chrome_options,
executable_path=executable_path)
This is the code (the crhomedriver execution parts):
from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.expected_conditions import presence_of_element_located
import os
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
log_path = os.path.join(BASE_DIR, 'cronJobChromeDriver.log')
executable_path = os.path.join(BASE_DIR, 'chromedriver_linux64/chromedriver')
extension_path = os.path.join(
BASE_DIR, 'chromedriver_linux64/extension_2_8_9_0.crx')
print('executable_path', executable_path)

The bottom line is, No, google-chrome-headless doesn't supports extensions.
In the one of his comment, alexclarke#chromium.org mentioned:
I realize a lot of folks would like to use extensions with headless but unfortunately that's a large project which we have /no plans to do/. The problem is Headless Chromium is a content embedder which means it doesn't have access to anything from other content embedders such as chrome and unfortunately extensions are a chrome feature.
In another comment he further added, if you're using Selenium through DevTools you can build a proxy. Next you can filter URLs and modify headers via Network.setRequestInterception and Network.continueInterceptedRequest.
Reference
You can find a relevant detailed discussion in:
Is it possible to run Google Chrome in headless mode with extensions?

this is now possible by modifying the following flag:
chrome_options.add_argument('--headless=chrome')
I tested it successfully.
I found it here: https://bugs.chromium.org/p/chromium/issues/detail?id=706008#c5

Chrome does not support headless, but apparently Firefox does.
Some relevant discussions:
https://sqa.stackexchange.com/questions/32611/selenium-chromedriver-headless-chrome-failed-to-wait-for-extension-backgro
Is it possible to run Google Chrome in headless mode with extensions?