I managed to get Python Selenium work with TOR in order to anon IP address, using the following code:
from selenium import webdriver
import os
torexe = os.popen(r'.\tor\tor.exe')
PROXY = "socks5://localhost:9050" # IP:PORT or HOST:PORT
options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=%s' % PROXY)
prefs = {"profile.managed_default_content_settings.images": 1,
"javascript.enabled": True}
options.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(options=options, executable_path=r'../../ChromeDriver.LENOVO/chromedriver.exe')
driver.get("https://asdfiles.com")
It is slow but that's ok: it is expected. And it works fine: with icanhazip.com I can check that I am browsing with another IP. HOWEVER, when I try to load https://asdfiles.com/ the page is not rendering correctly, as the screenshot below:
WHEN I LOAD IN TOR BROWSER, it renders correctly:
The problem is that there is a buttom a need to click using selenium, and when I do that nothing happens. I tried to inspect the code and found nothing. Tried to enable javascript on chromedriver and it did not work also.
Any suggestions how to tacke this?
I am running Python 3.8.8 on Windows 10;
When I run tor.exe it says: Tor 0.3.2.10 running on Windows 8 with Libevent 2.0.22-stable, OpenSSL 1.0.2n, Zlib 1.2.8, Liblzma N/A, and Libzstd N/A;
Chome 96.0.4664.45 (Versão oficial) 64 bits
Cheers!
Related
https://www.codegrepper.com/code-examples/python/python+selenium+brave+browser
I see this example to use brave browser on windows. Is it supposed to work on Catalina as well by just replacing driver_path and brave_path?
Also, Chromedriver is only for Chrome. How to determine which version of chromedriver should be used for brave browser?
https://chromedriver.chromium.org
from selenium import webdriver
driver_path = "C:/Users/username/PycharmProjects/chromedriver.exe"
brave_path = "C:/Program Files (x86)/BraveSoftware/Brave-Browser/Application/brave.exe"
option = webdriver.ChromeOptions()
option.binary_location = brave_path
# option.add_argument("--incognito") OPTIONAL
# option.add_argument("--headless") OPTIONAL
# Create new Instance of Chrome
browser = webdriver.Chrome(executable_path=driver_path, chrome_options=option)
browser.get("https://www.google.es")
Prerequisites:
Your chromedriver version should match your Brave Browser web driver version.
To make sure it does:
Check ChromeDriver version with brew info chromedriver. Along the lines of the output it should read chromedriver: 89.0.4389.23 (latest version as of writing this post)
Open Brave Browser, in menu bar click Brave -> About Brave. Along the lines it should read Version 1.22.71 Chromium: 89.0.4389.114 (Official Build) (x86_64) (again, latest as of writing this post)
These two should match, however, i am not entirely sure to which degree, since, as you can see here, last entries (.23 and .114) don't match, yet this works perfectly fine on my machine (macOS Big Sur 11.2.3) I don't think macOS version should really matter, but i still mentioned it for the sake of completeness.
Finally run the following code (replace paths with ones on your machine if they are different):
from selenium import webdriver
driverPath = '/usr/local/Caskroom/chromedriver/89.0.4389.23/chromedriver'
binaryPath = '/Applications/Brave Browser.app/Contents/MacOS/Brave Browser'
options = webdriver.ChromeOptions()
options.binary_location = binaryPath
browser = webdriver.Chrome(executable_path=driverPath, chrome_options=options)
browser.get("https://www.google.es")
If you have never used chromedriver before, after runnning the code you should see a macOS prompt saying that chromedriver is from unknown developer or was downloaded from the internet, smth like that. Close that prompt (It's important that you do this before moving on). Then go to System Preferences -> Security & Privacy -> press the lock icon and unlock it and then approve chromedriver to run on your machine. Run the above code again, a new macOS prompt will appear saying smth about unknown developer again, this time you can just click Open. Brave Browser window should pop up at this point. At least it did on my machine.
P.S. I apologise for possibly going into too much detail, but sometimes i get really frustrated with answers which skip parts which are considered to be obvious
For the next person looking, this is the most current way of using Brave with Selenium on Mac:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
driverPath = "/Applications/chromedriver" # Path to ChromeDriver
service = Service(driverPath)
options = webdriver.ChromeOptions()
options.binary_location = "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser" # Path to Brave Browser (this is the default)
driver = webdriver.Chrome(service=service, options=options)
# From here its Selenium as usual, example:
driver.get("https://google.com")
print(driver.title)
driver.close()
I'm trying to initiate a tor browsing session through Tor Browser 9.5 which uses the default Firefox v68.9.0esr using GeckoDriver and Selenium through Python on a windows-10 system. But I'm facing an error as:
Code Block:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import os
torexe = os.popen(r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Tor\tor.exe')
profile = FirefoxProfile(r'C:\Users\username\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
firefox_options = webdriver.FirefoxOptions()
firefox_options.binary_location = r'C:\Users\username\Desktop\Tor Browser\Browser\firefox.exe'
driver = webdriver.Firefox(firefox_profile= profile, options = firefox_options, executable_path=r'C:\WebDrivers\geckodriver.exe')
driver.get("https://www.tiktok.com/")
Where as the same code block works through Firefox and Firefox Nightly using the respective binaries.
Do I need any additional settings? Can someone help me out?
Firefox Snapshot:
Firefox Nightly Snapshot:
I managed to resolve this by updating to v9.5.1 and implementing the following changes:
Note that although the code is in C# the same changes to the Tor browser and how it is launched should be applied.
FirefoxProfile profile = new FirefoxProfile(profilePath);
profile.SetPreference("network.proxy.type", 1);
profile.SetPreference("network.proxy.socks", "127.0.0.1");
profile.SetPreference("network.proxy.socks_port", 9153);
profile.SetPreference("network.proxy.socks_remote_dns", false);
FirefoxDriverService firefoxDriverService = FirefoxDriverService.CreateDefaultService(geckoDriverDirectory);
firefoxDriverService.FirefoxBinaryPath = torPath;
firefoxDriverService.BrowserCommunicationPort = 2828;
var firefoxOptions = new FirefoxOptions
{
Profile = null,
LogLevel = FirefoxDriverLogLevel.Trace
};
firefoxOptions.AddArguments("-profile", profilePath);
FirefoxDriver driver = new FirefoxDriver(firefoxDriverService, firefoxOptions);
driver.Navigate().GoToUrl("https://www.google.com");
Important notes:
The following TOR configs need to be changed in about:config :
marionette.enabled: true
marionette.port: set to an unused port, and set this value to firefoxDriverService.BrowserCommunicationPort in your code. This was set to 2828 in my example.
note:
I am not sure whether this really is the definite answer (thus, I'd really appreciate feedback)
solution:
I've managed to send a get request to the check tor page (https://check.torproject.org/) and it displayed an unknown IP to me (additionally, IPs differ if you repeat the request after a time)
Essentially, I've set up the chrome driver to run TOR. Here's the code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
tor_proxy = "127.0.0.1:9150"
chrome_options = Options()
chrome_options.add_argument("--test-type")
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--disable-extensions')
chrome_options.add_argument('disable-infobars')
chrome_options.add_argument("--incognito")
chrome_options.add_argument('--proxy-server=socks5://%s' % tor_proxy)
driver = webdriver.Chrome(options=chrome_options)
driver.get('https://check.torproject.org/')
Because the driver is not in headless mode you can inspect the resulting page yourself. It should read:
"Congratulations. This browser is configured to use Tor. [IP Info]. However, it does not appear to be Tor Browser. Click here to go to the download page"
Make sure that the chromedriver.exe file is linked on the path or provide the path to the file as an argument to the driver.Chrome() function.
Edit: make sure TOR browser is running in the background, thanks #Abhishek Rai for pointing that out
I've successfully managed to load a Chrome Profile on MAC and I was trying to replicate the same on Linux but without success (Debian). I'm using Python, and the following works just fine on a MAC
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("user-data-dir=/Users/username/Library/Application Support/Google/Chrome")
driver = webdriver.Chrome('./chromedriver', options=chrome_options)
The same code on Debian, just doesn't work...
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--user-data-dir=/home/username/.config/google-chrome")
# I've tried also without the `--` but same outcome
# chrome_options.add_argument("user-data-dir=/home/username/.config/google-chrome")
driver = webdriver.Chrome('./chromedriver_linux', options=chrome_options)
I honestly now idea what's wrong. I'm using chromedriver 2.45 https://chromedriver.storage.googleapis.com/index.html?path=2.45/ and the issue is related to "Debian GNU/Linux 9 (stretch)" ...
In terms of launching Chrome, they both works. The difference is that on MAC it loads the profile, on Debian it doesn't.
Anyone has an idea why this is happening?
Right, so after many headaches, apparently this is something to do with the fact than I'm using CRD (Chrome Remote Desktop) to connect to the Linux instances!
In fact, you can check the profile location loading chrome://version. When connecting with CRD, this changes from the usual /home/user/.config/google-chrome to /home/user/.config/chrome-remote-desktop/chrome-profile/
All I needed to do is basically replace with the CRD directory to get all the profile information I wanted!
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
#chrome_options.add_argument("--user-data-dir=/home/user/.config/google-chrome")
chrome_options.add_argument("--user-data-dir=/home/user/.config/chrome-remote-desktop/chrome-profile/")
driver = webdriver.Chrome('./chromedriver_linux', options=chrome_options)
Hopefully this will be helpful for others! :)
I used to set up proxy on chrome like in a code below, but when i updated to selenium 3.8.1 proxy stops working, i dont get any errors it just doesn't use proxy server and i dont know why. My chromedriver is also up to date.
options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=192.99.55.120:3128')
driver = webdriver.Chrome(executable_path='C:\chromedriver_win32\chromedriver.exe', chrome_options=options)
driver.get("http://google.com/")
Would like to receive any advice, maybe alternative way to set up proxy for chromedriver.
If someone still interested, this is how i have finally solved the problem
from selenium.webdriver import Proxy
settings = {
"httpProxy": "192.99.55.120:3128",
"sslProxy": "192.99.55.120:3128"
}
proxy = Proxy(settings)
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
cap = DesiredCapabilities.CHROME.copy()
cap['platform'] = "WINDOWS"
cap['version'] = "10"
proxy.add_to_capabilities(cap)
from selenium.webdriver.chrome.webdriver import WebDriver as ChromeDriver
driver = ChromeDriver(desired_capabilities=cap, executable_path='C:\chromedriver_win32\chromedriver.exe')
try
options.add_argument('--proxy-server="http=192.99.55.120:3128;https=192.99.55.120:3128"')
also try running your chrome binary directly with these params to see whether it works or not
chrome.exe --proxy-server="http=192.99.55.120:3128"
If the navigator asks for the credentials username and password for the proxy and you need to handle this : (only if the alert come up)
driver.get("http://username:password#google.com/")
I am working on downloading HAR from Chrome for YouTube through Selenium Python Script.
Code Snippet:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--proxy-server={0}".format(url))
chrome_options.add_argument("--enable-quic")
self.driver = webdriver.Chrome(chromedriver,chrome_options = chrome_options)
self.proxy.new_har(args['url'], options={'captureHeaders': True})
self.driver.get(args['url'])
result = json.dumps(self.proxy.har, ensure_ascii=False)
I want QUIC to be used whenever I download HAR but when I look at the packets through Wireshark Selenium driver is using TCP only. Is there a way to force Chrome Driver to use QUIC? Or Is there an alternate to BMP?
A similar thing has been asked for Firefox in this question How to capture all requests made by page in webdriver? Is there any alternative to Browsermob? and there was a solution with Selenium alone without need of any BMP. So is it possible for Chrome?
Workaround for this problem could be: start Chrome normally (with your default profile or create another profile) and enable quic manually. Then start chromedriver with your profile loaded.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=/home/user/.config/google-chrome")
driver = webdriver.Chrome(executable_path="/home/user/Downloads/chromedriver", chrome_options=options)