i'm writing tests in selenium and tried to use only socks for proxy. Here is the code.
from selenium import webdriver
proxy = "localhost"
port = "5900"
profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.socks", proxy)
profile.set_preference("network.proxy.socks_port", port)
profile.update_preferences()
driver = webdriver.Firefox(executable_path="C:\Program Files\MozillaFirefox\geckodriver\geckodriver.exe", firefox_profile=profile)
driver.get("https://[xxx:xxx:xxx:xxx]")
But when I checked in Firefox. the proxy is set for all HTTP, FTP, etc instead of only Socks. Here is the image
Example
I know I am missing something Basic here. What is it?
profile.set_preference("network.proxy.socks_port", int(port))
I'm starting a new Django project, I'm trying to capture the network traffic with Selenium.
I have already achieved this objective with Selenium-wire (MITM Proxy), but Django doesn't like to work with selenium-wire ( must start the server with "--nothreading --noreload", connection bug... ).
I'm looking for achieve this with modern solutions, like parsing the network devtools of firefox directly or with a firefox addons.
I'm using Firefox Geckodriver for my test.
for x in range(0, 10):
profile = profile_firefox()
options = options_firefox()
driver = webdriver.Firefox(firefox_profile=profile, options=options, executable_path='/Users/*****/Desktop/selenium-master/headless_browser/geckodriver')
try:
driver.set_window_position(0, 0)
driver.set_window_size(randint(1024, 2060), randint(1024, 4100))
time.sleep(randint(3,10))
driver.get(url)
wait = ui.WebDriverWait(driver, 10)
time.sleep(randint(8,10))
if driver.find_element_by_xpath("//*[#id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button"):
driver.find_element_by_xpath("//*[#id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button").click()
del driver.requests
time.sleep(randint(8,10))
driver.find_element_by_xpath("//*[#id=\"container\"]/main/div/div/div[1]/div[2]/form/div/div[2]/div[1]/button").click()
time.sleep(randint(10,20))
for request in driver.requests:
if request.path == "https://api.*********.**/*******/*******":
request_api = request
raw = str(request_api.body)
request_api = raw.split(('b\''))
payload_raw = request_api[1]
payload = payload_raw[:-1]
if payload:
header = request.headers
time.sleep(8)
break
except:
print("Houston on a eu un probleme")
firefox_closing(driver)
Edit :
def profile_firefox():
profile = FirefoxProfile()
profile.set_preference('permissions.default.image', 2)
profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false')
profile.set_preference("general.useragent.override", firefox_init())
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', 'localhost')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.set_preference("driver.privatebrowsing.autostart", True)
profile.update_preferences()
return profile
Test 2 with Socks,HTTP,SSL configuration :
server = Server('/Users/*****/Desktop/selenium-master/browsermob-proxy-2.1.4/bin/browsermob-proxy')
server.start()
proxy = server.create_proxy()
proxy.selenium_proxy()#Dont understand what it does ???
port = int(proxy.port)
profile = FirefoxProfile()
profile.set_preference('permissions.default.image', 2)
profile.set_preference('dom.ipc.plugins.enabled.libflashplayer.so', 'false')
profile.set_preference('general.useragent.override', firefox_init())
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', 'localhost')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference('network.proxy.ssl', 'localhost')
profile.set_preference('network.proxy.ssl_port', port)
profile.set_preference('network.proxy.http', 'localhost')
profile.set_preference('network.proxy.http_port', port)
profile.set_preference('network.proxy.socks_remote_dns', False)
profile.set_preference('driver.privatebrowsing.autostart', True)
profile.update_preferences()
It seems Http proxy override the socks configuration...
Thanks a lot if you have any clue or advice about my code or solutions.
You can use a proxy to catch the network traffic. browsermob-proxy works well with selenium in Python. You need to download browsermob executable before. This is the piece of code with Firefox :
from browsermobproxy import Server
from selenium import webdriver
server = Server('path_to_executable')
server.start()
proxy = server.create_proxy()
profile = webdriver.FirefoxProfile()
profile.set_proxy(proxy.selenium_proxy())
driver = webdriver.Firefox(firefox_profile=profile)
proxy.new_har("file_name", options={'captureHeaders': True, 'captureContent': True})
driver.get("your_url")
proxy.wait_for_traffic_to_stop(1, 60)
for ent in proxy.har['log']['entries']:
print(ent)
server.stop()
driver.quit()
Python has a package called selenium-wire. You can use that package to capture the network traffics and also validate them. selenium-wire is an extended version of selenium will all the capabilities of selenium along with extra API to capture the network and validate. following is a link of an article
https://sensoumya94.medium.com/validate-network-requests-using-selenium-and-python-3da5be112f7b
following is the repository of the package
https://github.com/wkeeling/selenium-wire
Sample code -
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Firefox driver
driver = webdriver.Firefox()
# Go to the Google home page
driver.get('https://www.google.com')
# Access requests via the `requests` attribute
for request in driver.requests:
if request.response:
print(
request.url,
request.response.status_code,
request.response.headers['Content-Type']
)
Browsermob is the right way.
I must understand how browsermob works and tor too.
For Tor you must enable the HTTPTunnelPort configuration like this.
tor --HTTPTunnelPort 8088
And configure browsermob to use it.
proxy_params = {'httpProxy': 'localhost:8088', 'httpsProxy': 'localhost:8088'}
proxy_b = server.create_proxy(params=proxy_params)
Thanks.
I am trying to use Chrome along with python webdriver + selenium, but it seems not working when I set the proxy settings? Here is my code:
from selenium import webdriver
PROXY = 'http://42.115.88.220:53281'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
wbe = webdriver.Chrome(options=chromeOptions)
wbe.get("http://icanhazip.com")
When I run the above codes, the browser gives me: "This site can’t be reached" error:
This site can’t be reached
The connection was reset.
Try:
Checking the connection
Checking the proxy and the firewall
Running Windows Network Diagnostics
ERR_CONNECTION_RESET
Some Efforts: I tried requests with my proxy server, and it works. So it shouldn't be the problem of my proxy server.
import requests
proxies = {"http": "http://42.115.88.220:53281"}
r = requests.get("http://icanhazip.com", proxies = proxies)
print (r.status_code)
This gives me a response code of 200 and good response.
Goal: My final goal is to build a web-crawler with headless chrome with PROXY, so now I am testing a non-headless one first. But it seems there have been something wrong with this PROXY issue.
I would be really appreciated if anyone could help me out with this problem!!!
Try this.
For me it seems that you have used wrong type of headless mode. For chrome selenium browsers it's important to set --headless argument correct.
from selenium import webdriver
PROXY = 'http://ip:port'
chromeOptions = webdriver.ChromeOptions()
chromeOptions.add_argument('--proxy-server=%s' % PROXY)
chromeOptions.add_argument("ignore-certificate-errors")
# Headless mode for chrome browser
chromeOptions.add_argument('--headless=chrome')
wbe = webdriver.Chrome('your_driver_path_or_service', options=chromeOptions)
wbe.get("http://icanhazip.com")
print(wbe.title)
print(wbe.current_url)
print(wbe.page_source)
# Output:
# http://icanhazip.com/
# <html><head><meta name="color-scheme" content="light dark"></head><body><pre
# style="word-wrap: break-word; white-space: pre-wrap;">your ip
# </pre></body></html>
I run selenium WebDriver with a proxy.
chrome_options.add_argument('--proxy-server="'94.242.58.108:1448'"
driver = Chrome(chrome_options=chrome_options)
Driver know which proxy server it is using now.
Could you tell me how to find out the used proxy server?
I need this information in another function.
See the java doc http://seleniumhq.github.io/selenium/docs/api/java/org/openqa/selenium/Proxy.html
we have getHttpProxy() to retrieve the proxy.
Proxy p=new Proxy(); // used at the start to set proxy and pass to chrome options and driver
System.out.println(p.getHttpProxy()); // to get proxy when required.
I am having a strange issue. I have used the below code for a while to launch a anonymous wedriver instance, however somehow my settings of Chrome have refreshed and now the traffic is not going through the proxy. I tried making changes to the code and the chrome options but I can't figure out what I am doing wrong.
Also, it has somehow stopped adding the uBlock extension.
def anon_drive():
os.startfile(r"C:\Users\xxx\Downloads\tor-win32-0.2.9.9\Tor\tor.exe")
time.sleep(10)
chrome_path = r'C:\Users\xxx\Desktop\chromedriver.exe'
chop = webdriver.ChromeOptions()
chop.add_extension(r'C:\Users\Sid\Desktop\Ublock\uBlock-Origin_v1.11.0.crx')
driver = webdriver.Chrome(chrome_path, chrome_options=chop)
driver = webdriver.Chrome(chrome_path)
return driver
I have the Chrome options set to "Use a proxy server for your LAN is checked" I have tried setting the Advanced options to 127.0.0.1:9050 and 9051.
Still getting
Socks version 67 not recognised.(Tor is not an http proxy.)
I am not sure what I am doing wrong. There are some answers which I found but I am unable to find one which would perfectly.
#JamesKPolk gave me the answer.
Steps:
Chrome>Settings>Advanced Settings>Open Proxy Settings> LAN Settings>Use a Proxy Server for LAN> Advanced> Socks: 127.0.0.1 Port 9050
Press OK and everything should work.
Note: Leave everything except Socks empty.