I run selenium WebDriver with a proxy.
chrome_options.add_argument('--proxy-server="'94.242.58.108:1448'"
driver = Chrome(chrome_options=chrome_options)
Driver know which proxy server it is using now.
Could you tell me how to find out the used proxy server?
I need this information in another function.
See the java doc http://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/Proxy.html
we have getHttpProxy() to retrieve the proxy.
Proxy p=new Proxy(); // used at the start to set proxy and pass to chrome options and driver
System.out.println(p.getHttpProxy()); // to get proxy when required.
Related
I would like to make a program that modifies my ip when I go to consult a site with selenium, I use webdriver firefox, unfortunately the site that I use for my tests returns my ip and not the ip that I indicated in the options, could you tell me the error please.
The program launches and the firefox page opens (I don't use headless for the test), but it's my ip that is returned and not the one from the specified proxy.
here is my program
from selenium import webdriver
options = webdriver.FirefoxOptions()
proxy = f'{"137.74.65.101"}:{"80"}'
options.add_argument(f'--proxy-server={proxy}')
driver = webdriver.Firefox(options=options)
driver.get('https://httpbin.org/ip')
I recently made a script that allow me to create multiple Browser sessions targeting one URL. I would like to add proxy support to it in order to not get banned when running it. I tried to use the Proxy lib from selenium but it just get ignored.
My Question : How can I add proxy support into this script while using Selenium in python ? (each session will get a random proxy)
Here is my code
You could use the stem library which allows you to use Tor in python. Read the docs here to see how to use it.
The two basic parts missing from your code are the following:
from selenium.webdriver.common.proxy import Proxy, ProxyType
chrome_options.add_argument('--proxy-server=#yourproxyhere#'
Tor!
Here you can see how I set up my stem + selenium project:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.proxy import Proxy, ProxyType
from time import sleep
from stem import Signal
from stem.control import Controller
#this gives you a new identity
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM)
#set the proxy in selenium to 127.0.0.1:9150 and have your Tor Browser open!
link = 'https://some-link.com' #target url
prox= 'socks5://127.0.0.1:9150' #Here you connect to your localhost which connects to a Tor network
#some chrome_options
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % prox)
chrome_options.add_argument("--window-size=400,600")
#the following also deactivates location tracking!
prefs = {"profile.default_content_setting_values.geolocation" :2}
chrome_options.add_experimental_option("prefs",prefs)
driver = webdriver.Chrome("path_to_chromedriver", chrome_options=chrome_options)
driver.get(link)
Here is an updated version of my code including the new proxy support elements, I want it to use proxies form a txt file via a proxies = read_from_txt("proxies.txt") and randomly rotate between them using random lib :
Thank you for quick replies, really appreciate it.
I am trying to use Chrome along with python webdriver + selenium, but it seems not working when I set the proxy settings? Here is my code:
from selenium import webdriver
PROXY = 'http://42.115.88.220:53281'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
wbe = webdriver.Chrome(options=chromeOptions)
wbe.get("http://icanhazip.com")
When I run the above codes, the browser gives me: "This site can’t be reached" error:
This site can’t be reached
The connection was reset.
Try:
Checking the connection
Checking the proxy and the firewall
Running Windows Network Diagnostics
ERR_CONNECTION_RESET
Some Efforts: I tried requests with my proxy server, and it works. So it shouldn't be the problem of my proxy server.
import requests
proxies = {"http": "http://42.115.88.220:53281"}
r = requests.get("http://icanhazip.com", proxies = proxies)
print (r.status_code)
This gives me a response code of 200 and good response.
Goal: My final goal is to build a web-crawler with headless chrome with PROXY, so now I am testing a non-headless one first. But it seems there have been something wrong with this PROXY issue.
I would be really appreciated if anyone could help me out with this problem!!!
Try this.
For me it seems that you have used wrong type of headless mode. For chrome selenium browsers it's important to set --headless argument correct.
from selenium import webdriver
PROXY = 'http://ip:port'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
# Headless mode for chrome browser
chromeOptions.add_argument('--headless=chrome')
wbe = webdriver.Chrome('your_driver_path_or_service', options=chromeOptions)
wbe.get("http://icanhazip.com")
print(wbe.title)
print(wbe.current_url)
print(wbe.page_source)
# Output:
# http://icanhazip.com/
# <html><head><meta name="color-scheme" content="light dark"></head><body><pre
# style="word-wrap: break-word; white-space: pre-wrap;">your ip
# </pre></body></html>
I am having a strange issue. I have used the below code for a while to launch a anonymous wedriver instance, however somehow my settings of Chrome have refreshed and now the traffic is not going through the proxy. I tried making changes to the code and the chrome options but I can't figure out what I am doing wrong.
Also, it has somehow stopped adding the uBlock extension.
def anon_drive():
os.startfile(r"C:\Users\xxx\Downloads\tor-win32-0.2.9.9\Tor\tor.exe")
time.sleep(10)
chrome_path = r'C:\Users\xxx\Desktop\chromedriver.exe'
chop = webdriver.ChromeOptions()
chop.add_extension(r'C:\Users\Sid\Desktop\Ublock\uBlock-Origin_v1.11.0.crx')
driver = webdriver.Chrome(chrome_path, chrome_options=chop)
driver = webdriver.Chrome(chrome_path)
return driver
I have the Chrome options set to "Use a proxy server for your LAN is checked" I have tried setting the Advanced options to 127.0.0.1:9050 and 9051.
Still getting
Socks version 67 not recognised.(Tor is not an http proxy.)
I am not sure what I am doing wrong. There are some answers which I found but I am unable to find one which would perfectly.
#JamesKPolk gave me the answer.
Steps:
Chrome>Settings>Advanced Settings>Open Proxy Settings> LAN Settings>Use a Proxy Server for LAN> Advanced> Socks: 127.0.0.1 Port 9050
Press OK and everything should work.
Note: Leave everything except Socks empty.
I am using selenium with python.
I need to configure a proxy.
It is working for HTTP but not for HTTPS.
The code I am using is:
# configure firefox
profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", '11.111.11.11')
profile.set_preference("network.proxy.http_port", int('80'))
profile.update_preferences()
# launch
driver = webdriver.Firefox(firefox_profile=profile)
driver.get('https://www.iplocation.net/find-ip-address')
Also. Is there a way for me to completely block any outgoing traffic from my IP and restrict it ONLY to the proxy IP so that I don't accidently mess up the test/stats by accidently switching from proxy to direct connection?
Any tips would help!
Thanks :)
Check out browsermob proxy for setting up a proxies for use with selenium
from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
server.start()
proxy = server.create_proxy()
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("google")
driver.get("http://www.google.co.uk")
proxy.har # returns a HAR JSON blob
server.stop()
driver.quit()
You can use a remote proxy server with the RemoteServer class.
Is there a way for me to completely block any outgoing traffic from my IP and restrict it ONLY to the proxy IP
Yes, just look up how to setup proxies for whatever operating system you're using. Just use caution because some operating systems will ignore proxy rules based on certain conditions, for example, if using a VPN connection.