How to provide Username and Password to a proxy server in Firefox with selenium in python?
I tried the code below but the IP address doesn't change. I check the proxy setting in Firefox, it shows that only http proxy is changed, but not https.
proxy = {'host': '111.111.111.11',
'port': 8080,
'usr': USERNAME,
'pwd': PASSWORD
}
profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", proxy['host'])
profile.set_preference("network.proxy.https", proxy['host'])
profile.set_preference("network.proxy.http_port", proxy['port'])
profile.set_preference("network.proxy.username", proxy['usr'])
profile.set_preference("network.proxy.password", proxy['pwd'])
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile=profile)
driver.get('https://whatismyipaddress.com')
I see a solution for ChromeDriver (but I don't test it because I can only use Firefox):
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=196.19.8.80:8000')
chrome_options.add_argument('--proxy-auth=yjd5Rn:29apJZ')
chrome = webdriver.Chrome(chrome_options=chrome_options)
chrome.get("http://whatismyipaddress.com")
As you can see, ChromeOptions have '--proxy-auth=yjd5Rn:29apJZ' argument.
Could anyone tell me how to use proxy and provide authorization to proxy server in Firefox with Selenium and in python? Thanks in advance!
Below is the login page from the proxy server:
I'm starting a new Django project, I'm trying to capture the network traffic with Selenium.
I have already achieved this objective with Selenium-wire (MITM Proxy), but Django doesn't like to work with selenium-wire ( must start the server with "--nothreading --noreload", connection bug... ).
I'm looking for achieve this with modern solutions, like parsing the network devtools of firefox directly or with a firefox addons.
I'm using Firefox Geckodriver for my test.
for x in range(0, 10):
profile = profile_firefox()
options = options_firefox()
driver = webdriver.Firefox(firefox_profile=profile, options=options, executable_path='/Users/*****/Desktop/selenium-master/headless_browser/geckodriver')
try:
driver.set_window_position(0, 0)
driver.set_window_size(randint(1024, 2060), randint(1024, 4100))
time.sleep(randint(3,10))
driver.get(url)
wait = ui.WebDriverWait(driver, 10)
time.sleep(randint(8,10))
if driver.find_element_by_xpath("//*[#id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button"):
driver.find_element_by_xpath("//*[#id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button").click()
del driver.requests
time.sleep(randint(8,10))
driver.find_element_by_xpath("//*[#id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button").click()
time.sleep(randint(10,20))
for request in driver.requests:
if request.path == "https://api.*********.**/*******/*******":
request_api = request
raw = str(request_api.body)
request_api = raw.split(('b\''))
payload_raw = request_api[1]
payload = payload_raw[:-1]
if payload:
header = request.headers
time.sleep(8)
break
except:
print("Houston on a eu un probleme")
firefox_closing(driver)
Edit :
def profile_firefox():
profile = FirefoxProfile()
profile.set_preference('permissions.default.image', 2)
profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false')
profile.set_preference("general.useragent.override", firefox_init())
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', 'localhost')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.set_preference("driver.privatebrowsing.autostart", True)
profile.update_preferences()
return profile
Test 2 with Socks,HTTP,SSL configuration :
server = Server('/Users/*****/Desktop/selenium-master/browsermob-proxy-2.1.4/bin/browsermob-proxy')
server.start()
proxy = server.create_proxy()
proxy.selenium_proxy()#Dont understand what it does ???
port = int(proxy.port)
profile = FirefoxProfile()
profile.set_preference('permissions.default.image', 2)
profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false')
profile.set_preference('general.useragent.override', firefox_init())
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', 'localhost')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference('network.proxy.ssl', 'localhost')
profile.set_preference('network.proxy.ssl_port', port)
profile.set_preference('network.proxy.http', 'localhost')
profile.set_preference('network.proxy.http_port', port)
profile.set_preference('network.proxy.socks_remote_dns', False)
profile.set_preference('driver.privatebrowsing.autostart', True)
profile.update_preferences()
It seems Http proxy override the socks configuration...
Thanks a lot if you have any clue or advice about my code or solutions.
You can use a proxy to catch the network traffic. browsermob-proxy works well with selenium in Python. You need to download browsermob executable before. This is the piece of code with Firefox :
from browsermobproxy import Server
from selenium import webdriver
server = Server('path_to_executable')
server.start()
proxy = server.create_proxy()
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("file_name", options={'captureHeaders': True, 'captureContent': True})
driver.get("your_url")
proxy.wait_for_traffic_to_stop(1, 60)
for ent in proxy.har['log']['entries']:
print(ent)
server.stop()
driver.quit()
Python has a package called selenium-wire. You can use that package to capture the network traffics and also validate them. selenium-wire is an extended version of selenium will all the capabilities of selenium along with extra API to capture the network and validate. following is a link of an article
https://sensoumya94.medium.com/validate-network-requests-using-selenium-and-python-3da5be112f7b
following is the repository of the package
https://github.com/wkeeling/selenium-wire
Sample code -
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Firefox driver
driver = webdriver.Firefox()
# Go to the Google home page
driver.get('https://www.google.com')
# Access requests via the `requests` attribute
for request in driver.requests:
if request.response:
print(
request.url,
request.response.status_code,
request.response.headers['Content-Type']
)
Browsermob is the right way.
I must understand how browsermob works and tor too.
For Tor you must enable the HTTPTunnelPort configuration like this.
tor --HTTPTunnelPort 8088
And configure browsermob to use it.
proxy_params = {'httpProxy': 'localhost:8088', 'httpsProxy': 'localhost:8088'}
proxy_b = server.create_proxy(params=proxy_params)
Thanks.
I am trying to use Chrome along with python webdriver + selenium, but it seems not working when I set the proxy settings? Here is my code:
from selenium import webdriver
PROXY = 'http://42.115.88.220:53281'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
wbe = webdriver.Chrome(options=chromeOptions)
wbe.get("http://icanhazip.com")
When I run the above codes, the browser gives me: "This site can’t be reached" error:
This site can’t be reached
The connection was reset.
Try:
Checking the connection
Checking the proxy and the firewall
Running Windows Network Diagnostics
ERR_CONNECTION_RESET
Some Efforts: I tried requests with my proxy server, and it works. So it shouldn't be the problem of my proxy server.
import requests
proxies = {"http": "http://42.115.88.220:53281"}
r = requests.get("http://icanhazip.com", proxies = proxies)
print (r.status_code)
This gives me a response code of 200 and good response.
Goal: My final goal is to build a web-crawler with headless chrome with PROXY, so now I am testing a non-headless one first. But it seems there have been something wrong with this PROXY issue.
I would be really appreciated if anyone could help me out with this problem!!!
Try this.
For me it seems that you have used wrong type of headless mode. For chrome selenium browsers it's important to set --headless argument correct.
from selenium import webdriver
PROXY = 'http://ip:port'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
# Headless mode for chrome browser
chromeOptions.add_argument('--headless=chrome')
wbe = webdriver.Chrome('your_driver_path_or_service', options=chromeOptions)
wbe.get("http://icanhazip.com")
print(wbe.title)
print(wbe.current_url)
print(wbe.page_source)
# Output:
# http://icanhazip.com/
# <html><head><meta name="color-scheme" content="light dark"></head><body><pre
# style="word-wrap: break-word; white-space: pre-wrap;">your ip
# </pre></body></html>
I am trying to connect to a proxy, however the browser is not using it and when I check my ip, it is my own ip and not the proxy.
I provided a fake IP, as my IP is dedicated and private.
here is what I've tried:
chrome_options = Options()
chrome_options.add_argument('--proxy=https://23.23.23.23:19191'))
chrome_options.add_argument('--proxy-auth=user:password')
chrome_options.add_argument('--proxy-type=https')
driver = webdriver.Chrome(chrome_options = chrome_options)
driver.get("http://whatismyipaddress.com/")
Also, how would i do this on phantomjs?
I am using selenium with python.
I need to configure a proxy.
It is working for HTTP but not for HTTPS.
The code I am using is:
# configure firefox
profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", '11.111.11.11')
profile.set_preference("network.proxy.http_port", int('80'))
profile.update_preferences()
# launch
driver = webdriver.Firefox(firefox_profile=profile)
driver.get('https://www.iplocation.net/find-ip-address')
Also. Is there a way for me to completely block any outgoing traffic from my IP and restrict it ONLY to the proxy IP so that I don't accidently mess up the test/stats by accidently switching from proxy to direct connection?
Any tips would help!
Thanks :)
Check out browsermob proxy for setting up a proxies for use with selenium
from browsermobproxy import Server
server = Server("path/to/browsermob-proxy")
server.start()
proxy = server.create_proxy()
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("google")
driver.get("http://www.google.co.uk")
proxy.har # returns a HAR JSON blob
server.stop()
driver.quit()
You can use a remote proxy server with the RemoteServer class.
Is there a way for me to completely block any outgoing traffic from my IP and restrict it ONLY to the proxy IP
Yes, just look up how to setup proxies for whatever operating system you're using. Just use caution because some operating systems will ignore proxy rules based on certain conditions, for example, if using a VPN connection.