Scraper automatically exits - python

I am creating a LinkedIn scraper using Selenium. My script executes properly but it gets close after 3-5 secs and it does not give any error in the console. It says Process finished with exit code 0
Here is my code:
import time,os
try:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support import expected_conditions as EC
except:
os.system("python -m pip install --upgrade pip")
os.system("pip3 install selenium")
os.system("pip3 install webdriver-manager")
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument('disable-infobars')
driver = webdriver.Chrome(ChromeDriverManager().install(),options=options)
driver.get("https://www.linkedin.com/")

Related

Python executable_path keyword no longer recognized when trying to launch chrome driver [duplicate]

I am using sublime to code python scripts. The following code is for selenium in python to install the driver automatically by using the webdriver_manager package
# pip install webdriver-manager
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
#s=Service(path)
#driver=webdriver.Chrome(service=s)
driver.get('https://www.google.com')
driver.find_element(By.NAME, 'q').send_keys('Yasser Khalil')
The code works fine but I got a warning like that
Demo.py:7: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
driver = webdriver.Chrome(ChromeDriverManager().install())
How to fix such a bug?
This error message...
DeprecationWarning: executable_path has been deprecated, please pass in a Service object
...implies that the key executable_path will be deprecated in the upcoming releases.
This change is inline with the Selenium 4.0 Beta 1 changelog which mentions:
Deprecate all but Options and Service arguments in driver instantiation. (#9125,#9128)
Solution
With selenium4 as the key executable_path is deprecated you have to use an instance of the Service() class along with ChromeDriverManager().install() command as discussed below.
Pre-requisites
Ensure that:
Selenium is upgraded to v4.0.0
pip3 install -U selenium
Webdriver Manager for Python is installed
pip3 install webdriver-manager
You can find a detailed discussion on installing Webdriver Manager for Python in ModuleNotFoundError: No module named 'webdriver_manager' error even after installing webdrivermanager
Selenium v4 compatible Code Block
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get("https://www.google.com")
Console Output:
[WDM] - ====== WebDriver manager ======
[WDM] - Current google-chrome version is 96.0.4664
[WDM] - Get LATEST driver version for 96.0.4664
[WDM] - Driver [C:\Users\Admin\.wdm\drivers\chromedriver\win32\96.0.4664.45\chromedriver.exe] found in cache
You can find a detailed discussion on installing Webdriver Manager for Python in Selenium ChromeDriver issue using Webdriver Manager for Python
Incase you want to pass the Options() object you can use:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
options = Options()
options.add_argument("start-maximized")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get("https://www.google.com")
TL; DR
You can find the relevant Bug Report/Pull Request in:
Bug Report: deprecate all but Options and Service arguments in driver instantiation
Pull Request: deprecate all but Options and Service arguments in driver instantiation
This works for me
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
ser = Service(r"C:\chromedriver.exe")
op = webdriver.ChromeOptions()
s = webdriver.Chrome(service=ser, options=op)
Extending on the accepted answer, the Service class allows to explicitly specify a ChromeDriver executable in the same way as previously using the executable_path parameter. In this way existing code is easily migrated (clearly you need to replace C:\chromedriver.exe above by your path).
I could figure it out
# pip install webdriver-manager
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
s=Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=s)
driver.maximize_window()
driver.get('https://www.google.com')
driver.find_element(By.NAME, 'q').send_keys('Yasser Khalil')
I found this deprecation issue is appearing on Selenium, Pip and Python updates. so simply just change :
before:
from selenium import webdriver
chrome_driver_path = 'C:/Users/Morteza/Documents/Dev/chromedriver.exe'
driver = webdriver.Chrome(executable_path=chrome_driver_path)
url = "https://www.google.com"
driver.get(url)
after:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
s=Service('C:/Users/Morteza/Documents/Dev/chromedriver.exe')
browser = webdriver.Chrome(service=s)
url='https://www.google.com'
browser.get(url)
All the above answers refer to Chrome, adding the one for Firefox
Install:
pip install webdriver-manager
Code:
from selenium.webdriver.firefox.service import Service
from webdriver_manager.firefox import GeckoDriverManager
driver = webdriver.Firefox(service=Service(executable_path=GeckoDriverManager().install()))
Reference: https://github.com/SergeyPirogov/webdriver_manager/issues/262#issuecomment-955197860
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
service_obj = Service("WebDrivers_path\chromedriver.exe")
driver = webdriver.Chrome(service=service_obj)
driver.get("https://www.google.com")
Simplest option with Chrome auto-installer:
from selenium import webdriver
import chromedriver_autoinstaller
from selenium.webdriver.chrome.service import Service
chromedriver_autoinstaller.install()
driver = webdriver.Chrome(service=Service())
Have a look at the new definition in the Service object here.
My solution
from selenium.webdriver.chrome.service import Service
chrome_executable = Service(executable_path='chromedriver.exe', log_path='NUL')
driver = webdriver.Chrome(service=chrome_executable)
if you are using any IDE like PyCharm install webdriver-manager package of that IDE as how do install for selenium package
You can create an instance of ChromeOptions, which has convenient methods for setting ChromeDriver-specific capabilities. You can then pass the ChromeOptions object into the ChromeDriver constructor:
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File("/path/to/extension.crx"));
ChromeDriver driver = new ChromeDriver(options);
Since Selenium version 3.6.0, the ChromeOptions class in Java also implements the Capabilities interface, allowing you to specify other WebDriver capabilities not specific to ChromeDriver.
ChromeOptions options = new ChromeOptions();
// Add the WebDriver proxy capability.
Proxy proxy = new Proxy();
proxy.setHttpProxy("myhttpproxy:3337");
options.setCapability("proxy", proxy);
// Add a ChromeDriver-specific capability.
options.addExtensions(new File("/path/to/extension.crx"));
ChromeDriver driver = new ChromeDriver(options);

Selenium launch browser in specific path (for example ./firefox/firefox.exe) (Python3)

I would like to do what is in the title.
My code:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('https://www.google.com')
Have you tried webdriver-manager library?
It's amazing, you just need to install it:
pip install webdriver-manager
and launch your code like this:
from selenium import webdriver
from webdriver_manager.firefox import GeckoDriverManager
driver = webdriver.Firefox(executable_path=GeckoDriverManager().install())
that package downloads you needed geckodriver automatically. Don't forget path 'executable_path' while launching driver.
As well you can you chromedriver, like that:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
Selenium searches the webdriver through the directories on sys.path, so you would first have to do something like that:
import sys
from selenium import webdriver
sys.path.insert(0,'/path/to/firefox')
driver = webdriver.Firefox()
driver.get('https://www.google.com')

I have already installed webdriver_manager but python can't find the module

This is from my script:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome('C:\\Users\\tques\\chromedriver_win32\\chromedriver.exe')
driver = webdriver.Chrome(ChromeDriverManager().install())
I have tried both instances of the "driver" variable, both independently and together but I always get this error "ModuleNotFoundError: No module named 'webdriver_manager'"
I've already tried uninstalling selenium and webdriver_manager, then reinstalling them, but the issue persists.
find the file path ware webdriver_manager is installed, then add
import sys
sys.path.append(<webdriver_manager path>)
before
from webdriver_manager.chrome import ChromeDriverManager
Have you tried to add a driver to the PATH? - HOW TO DO THIS
And then do as simple as:
from selenium import webdriver
browser = webdriver.Chrome()
try this:
from selenium import webdriver<br>
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())

Selenium: This version of ChromeDriver only supports Chrome version 83

I just had this code working last night but now it won't work and I am stumped. I am using ChromeDriverManager to ensure the right driver is being downloaded, but I still get the error. I have also tried removing the driver cache, but this doesn't help.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from webdriver_manager.chrome import ChromeDriverManager
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.implicitly_wait(20)
driver.get("google.com")
From this code I get the error
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 83
Edit: I am using Chrome Version 83.0.4103.97 and the ChromeDriver version being downloaded is 83.0.4103.39

Python Launching Chromedriver from selenium works in IDLE but not Visual Studio Code

Below is my code and when I run it through powershell/cmd/IDLE it launches the chromedriver perfectly. However when I try to run it through visual studio code I receive the text below.
"selenium.common.exceptions.WebDriverException: Message: 'chromedriver.exe' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home"
#SELENIUM INFO#
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
#SELENIUM INFO#
import os
print(os.getcwd())
driver = webdriver.Chrome(os.getcwd() +'\\chromedriver.exe')
I right click the code and click 'Run Python File in Terminal' However I've discovered that I can't even use the terminal to cd to the code deeper inside the file structure.
The file structure I have is "G:\Quality User Data\Malahy\Projects\AQE Interfaces>"
But the chrome driver and the python file are stored together in a deeper directory.
"G:\Quality User Data\Malahy\Projects\AQE Interfaces\AQE Interface\Logistics\Repack"
I'm starting to think this is a bug with Visual Studio Code more than my program.

Categories