from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=125.62.213.229:82')
browser = webdriver.Chrome(executable_path='F:\Drivers\ChromeDriver\chromedriver.exe', chrome_options=options)
browser.get("http://www.whatismyipaddress.com")
I dont know to apply loop in this.
using .format to substitute.
ip=['125.62.213.229:82'] # add others here.
for i in ip:
options.add_argument(('--proxy-server={0}').format(i))
#rest of your code
Related
I'm doing an automation using Selenium in Python
How could I simply ignore an error that happens on the site without it closing the browser?
I looked in several places about and I didn't find anything that helped me
You can do:
# Needed libs
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
driver = webdriver.Chrome(options=chrome_options)
Then, if something fails, your browser will not be closed.
I want to scrap some data with selenium python. I have this type of screen sometimes :
Do you know to proceed in order to remove this type of verification ? Here my code :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
options = Options()
options.add_argument("start-maximized")
options.add_argument("--disable-web-security")
options.add_argument("--disable-site-isolation-trials")
options.add_argument("--allow-running-insecure-content")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
driver.get('THE_WEBSITE_COM')
Selenium specifically and other automation tools have certain user agents and other identifiers which indicate that it's automated. So maybe have a play around with things like that. Some websites use anti bot tools to analyze browsing behaviours and patterns so try to slow it and randomize it eg. random time between page requests
Another trick is to look at the website and try to find if there are any alternative routes to get the information. For example: is there a public API you can use to bypass it? Is there a mobile version of the website? Sometimes mobile versions have less aggressive Captcha enforcement.
What I do most of the time is to launch the browser separately and connect to it using Dev port which is beautifully explained in this article.
To enable Chrome to open a port for remote debugging, we need to launch it with a custom flag –
chrome.exe --remote-debugging-port=9222 --user-data-dir="C:\selenum\ChromeProfile"
Then connect to the browser using this
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
#Change chrome driver path accordingly
chrome_driver = "C:\chromedriver.exe"
driver = webdriver.Chrome(chrome_driver, chrome_options=chrome_options)
print driver.title
I am using python selenium for web scraping, but after running the below codes, chrome is launched but did not get the website as I want, instead, it shows 'data;' in url bar.
Could anyone help with the problem? Many thanks!!
PS: My chrome is 88.and chromedriver is also 88. the path of chrome and chromedriver are different, one is in desktop and the other is C://
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
import random
option = Options()
option.add_experimental_option('excludeSwitches', ['enable-logging'])
option.add_argument('--remote-debugging-port=9222')
driver =webdriver.Chrome(executable_path='C:/Users/Desktop/chromedriver.exe')
driver.get("https://www.youtube.com")
Instead of using this
from selenium.webdriver.chrome.options import Options
options = options()
use this
options = webdriver.ChromeOptions()
and I would suggest you to put everything in the same project directory. which is a best practice.
Once you've put the chromedriver in the same directory.
do this:
from os import getcwd
driver = webdriver.Chrome(getcwd() + "\chromedriver.exe", options=options)
Also, try to close the old chrome windows.
It will go to data:// and after that it will redirect to your website.
And also follow Jiya's answer as well.
Im trying to make a bot that visits my adfly link using the chrome webdriver. Every time I try to use the code below though, the webdriver tells me that there were too many redirects and doesn't follow through. The code below is just being used for testing at the moment:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--proxy-server="+"http://102.129.249.120:8080")
browser = webdriver.Chrome(options=options)
browser.get("http://raboninco.com/18Whc")
Image of error here
Okay so i figured it out. I can use tor with selenium to get access to adfly. Works great btw. Thanks for the help and time guys. If you want to see the code I used, here it is:
from selenium import webdriver
import os
os.popen(r'C:\Users\joldb\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
options = webdriver.ChromeOptions()
options.add_argument("--proxy-server="+"socks5://127.0.0.1:9050")
browser = webdriver.Chrome(options=options)
browser.get("http://raboninco.com/18Whc")
I am working on downloading HAR from Chrome for YouTube through Selenium Python Script.
Code Snippet:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--proxy-server={0}".format(url))
chrome_options.add_argument("--enable-quic")
self.driver = webdriver.Chrome(chromedriver,chrome_options = chrome_options)
self.proxy.new_har(args['url'], options={'captureHeaders': True})
self.driver.get(args['url'])
result = json.dumps(self.proxy.har, ensure_ascii=False)
I want QUIC to be used whenever I download HAR but when I look at the packets through Wireshark Selenium driver is using TCP only. Is there a way to force Chrome Driver to use QUIC? Or Is there an alternate to BMP?
A similar thing has been asked for Firefox in this question How to capture all requests made by page in webdriver? Is there any alternative to Browsermob? and there was a solution with Selenium alone without need of any BMP. So is it possible for Chrome?
Workaround for this problem could be: start Chrome normally (with your default profile or create another profile) and enable quic manually. Then start chromedriver with your profile loaded.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=/home/user/.config/google-chrome")
driver = webdriver.Chrome(executable_path="/home/user/Downloads/chromedriver", chrome_options=options)