I'm trying to use proxy in selenium by this way:
firefox_options.add_argument('--proxy-server=socks5://' + 'username:pwd:addr#ipvanish.com:port')
I'm using ipvanish's proxies
there are some hostnames which I don't know where to put
Can someone please show me how can I configure a proxy properly, if possible, without using any extension? I searched and found some ways, though I was not able to comprehend the method yet so I'd appreciate it if you'd explain it explicitly.
If you are using user:pass authentication then you have to use Proxy Auto Auth extension for authentication.
Use Crx downloader extension for downloading extension as a zip file.
Then you can load the Proxy Auto Auth extension in the selenium browser.
For Python
import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
executable_path = "path_to_webdriver"
os.environ["webdriver.chrome.driver"] = executable_path
chrome_options = Options()
chrome_options.add_extension('path_to_extension')
chrome_options.add_argument('--proxy-server=socks5://' + 'addr#ipvanish.com:port')
driver=webdriver.Chrome(executable_path=executable_path,chrome_options=chrome_options)
driver.get("http://stackoverflow.com")
driver.quit()
Path to extension is the path to a folder containing manifest.json file of the extension
Then you can simply use the code auth the proxy following the below code
driver.Navigate().GoToUrl(#"chrome-extension://extensionName/options.html");
WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(5));
wait.Until(ExpectedConditions.ElementIsVisible(By.Id("login"))).Clear();
wait.Until(ExpectedConditions.ElementIsVisible(By.Id("login"))).SendKeys("username");
wait.Until(ExpectedConditions.ElementIsVisible(By.Id("password"))).Clear();
wait.Until(ExpectedConditions.ElementIsVisible(By.Id("password"))).SendKeys("pwd");
wait.Until(ExpectedConditions.ElementIsVisible(By.Id("retry"))).Clear();
wait.Until(ExpectedConditions.ElementIsVisible(By.Id("retry"))).SendKeys("5");
wait.Until(ExpectedConditions.ElementIsVisible(By.Id("save"))).Click();
This code for C# I hope you can translate for python.
Also, you can extract the downloaded Zip file of the extension and load it in the browser from the folder.
Related
I'm trying to capture all network logs using seleniumwire. When chromedriver is in normal mode, it is able to capture all requests. But when it is in headless mode, it is not capturing all requests.
I tried adding sleep(10), assert driver.last_request.response.status_code == 200
but neither helped.
Since seleniumwire is not that popular, I'm adding a sample guide below in the hope of getting people with knowledge of selenium to try a hand to help me fix the problem.
Working with seleniumwire
Installing seleniumwire
pip install seleniumwire
Sample script:
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Chrome driver
driver = webdriver.Chrome()
# Go to the YouTube homepage.
driver.get('https://www.youtube.com')
# Access requests via the `requests` attribute
for request in driver.requests:
if request.response:
print(
request.path,
request.response.status_code,
request.response.headers['Content-Type']
)
try to capture all requests
options = {
'ignore_http_methods': [] # Capture all requests, including OPTIONS requests
}
driver = webdriver.Chrome("C:\chromedriver.exe",seleniumwire_options=options)
In default it ignores OPTIONS method
When chrome browser is opened by selenium, it uses it's own profile rather than the default one present. Try using custom profile, for chrome you can use ChromeOptions class use a custom profile and try.
I wish to do a direct download of a PDF and not display in Chrome's pdf view plugin
The Python code I found is
chromeOptions = webdriver.ChromeOptions()
prefs = {"plugins.plugins_disabled" : ["Chrome PDF Viewer"]}
chromeOptions.add_experimental_option("prefs",prefs)
driver=webdriver.Chrome('/usr/lib/chromium-browser/chromedriver', chrome_options=chromeOptions)
chromeOptions does not have an add_experimental_option function/methodP.
Is there a way to make this work please.
Here is the proper way to initialize chrome options:
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
I believe that is your issue. I tested this code and it worked for me:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
prefs = {"plugins.plugins_disabled" : ["Chrome PDF Viewer"]}
chrome_options.add_experimental_option("prefs",prefs)
driver=webdriver.Chrome(chrome_options=chrome_options)
For more information you can read the docs here regarding the Chrome WebDriver API for Selenium
For whatever reason the method add_experimental_option does not appear. Possibly this is because I am using a Linux install. My goal is to download a series of PDFs automatically. A work around is to first get the PDF in the pdf-viewer by finding a web element with the click() command. this loads the PDF into the viewer, then read the contents of the URL bar, the use the PDF address to make a call to the Linux operating system running the dowload command "wget" to obtain the PDF file. That is:
driver.find_element_by_class_name('browzine-direct-to-pdf-link').click()
pdfAddress=driver.current_url
os.system("wget %s -P /home/keir/Downloads/pdfs" % pdfAddress)
I want to be able to use extensions (mainly chropath) while testing using selenium web driver. How can I set up my script to load extensions by default? It currently opens a page with no extensions enabled and does not remember if I enable extensions during a session.
Thank yoU!
Try the following code in Python.
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.desired_capabilities import.
DesiredCapabilities'
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File("/pathtoChromeextension.crx"));
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(ChromeOptions.CAPABILITY, options);
ChromeDriver driver = new ChromeDriver(capabilities);
You can use this to get crx file http://crxextractor.com/ from your extension id.
Is it possible to connect selenium to the browser I use normally instead a driver? For normal browsing I am using chrome with several plugins - add block plus, flashblock and several more. I want to try to load a site using this specific configuration. How can I do that?
p.s - I dont want to connect only to an open browser like in this question :
How to connect to an already open browser?
I dont care if I spawn the process using a driver. I just want the full browser configuration - cookies,plugins,fonts etc.
Thanks
First, you need to download the ChromeDriver, then either put the path to the executeable to the PATH environment variable, or pass the path in the executable_path argument:
from selenium import webdriver
driver = webdriver.Chrome(executable_path='/path/to/executeable/chrome/driver')
In order to load extensions, you would need to set ChromeOptions:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_extension('Adblock-Plus_v1.4.1.crx')
driver = webdriver.Chrome(chrome_options=options)
You can also save the chrome user profile you have and load it to the ChromeDriver:
options = webdriver.ChromeOptions()
options.add_argument('--user-data-dir=/path/to/my/profile')
driver = webdriver.Chrome(chrome_options=options)
See also:
Running Selenium WebDriver using Python with extensions (.crx files)
ChromeDriver capabilities/options
I am currently using Selenium to run instances of Chrome to test web pages. Each time my script runs, a clean instance of Chrome starts up (clean of extensions, bookmarks, browsing history, etc). I was wondering if it's possible to run my script with Chrome extensions. I've tried searching for a Python example, but nothing came up when I googled this.
You should use Chrome WebDriver options to set a list of extensions to load. Here's an example:
import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
executable_path = "path_to_webdriver"
os.environ["webdriver.chrome.driver"] = executable_path
chrome_options = Options()
chrome_options.add_extension('path_to_extension')
driver = webdriver.Chrome(executable_path=executable_path, chrome_options=chrome_options)
driver.get("http://stackoverflow.com")
driver.quit()
Hope that helps.
The leading answer didn't work for me because I didn't realize you had to point the webdriver options toward a .zip file.
I.e. chrome_options.add_extension('path_to_extension_dir') doesn't work.
You need: chrome_options.add_extension('path_to_extension_dir.zip')
After figuring that out and reading a couple posts on how to create the zip file via the command line and load it into selenium, the only way it worked for me was to zip my extension files within the same python script. This actually turned out to be a nice way for automatically updating any changes you might have made to your extension:
import os, zipfile
from selenium import webdriver
# Configure filepaths
chrome_exe = "path/to/chromedriver.exe"
ext_dir = 'extension'
ext_file = 'extension.zip'
# Create zipped extension
## Read in your extension files
file_names = os.listdir(ext_dir)
file_dict = {}
for fn in file_names:
with open(os.path.join(ext_dir, fn), 'r') as infile:
file_dict[fn] = infile.read()
## Save files to zipped archive
with zipfile.ZipFile(ext_file), 'w') as zf:
for fn, content in file_dict.iteritems():
zf.writestr(fn, content)
# Add extension
chrome_options = webdriver.ChromeOptions()
chrome_options.add_extension(ext_file)
# Start driver
driver = webdriver.Chrome(executable_path=chrome_exe, chrome_options=chrome_options)
driver.get("http://stackoverflow.com")
driver.quit()
This is because the selenium expects the packed extensions as the extension argument which will be with .crx extension.
I already had the extension in my chrome. Below are the steps I followed to pack the existing extension,
Click on your extension 'details'. In my Chrome version, it was on right top click (3 dots) -> 'More tools' -> 'Extensions'.
Have the developer mode enabled in your chrome
Click 'Pack extension' (As shown above) and pack it.
This will get stored in the extensions location. For me it was on /home/user/.config/google-chrome/Default/Extensions/fdjsidgdhskifcclfjowijfwidksdj/2.3.4_22.crx
That's it, you can configure the extension in your selenium as argument.
extension='/home/user/.config/google-chrome/Default/Extensions/fdjsidgdhskifcclfjowijfwidksdj/2.3.4_22.crx'
options = webdriver.ChromeOptions()
options.add_extension(extension)
NOTE: You can also find the 'fdjsidgdhskifcclfjowijfwidksdj' id in the extensions url as query param
If you wanna import any chrome extension in your selenium python scrip
Put your extension.crx.crx file in the same folder as your code or give the path
you can copy-paste this code and just change the file crx.crx name
import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
executable_path = "/webdrivers"
os.environ["webdriver.chrome.driver"] = executable_path
chrome_options = Options()
chrome_options.add_extension(' YOUR - EXTIONTION - NAME ')
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get("http://stackoverflow.com")
if this code is throwing an error maybe this will solve it
An alternative way is to use the unpacked folder method. I found the crx and zip method did not work at all on Selenium Grid. In this method, you need to copy the extension from the user data in a Chrome version where you have already manually installed it, the long ID made up of loads of letters like pehaalcefcjfccdpbckoablngfkfgfgj, to the user data directory of the Selenium-controlled Chrome (which you can choose at runtime using the first line of this code, and it will get populated automatically). It should be in the same equivalent directory (which is Extensions). The path must take you all the way to the directory where there is a manifest.json, hence in this example '1.1.0'
chrome_options.add_argument("user-data-dir=C:/Users/charl/OneDrive/python/userprofile/profilename"
unpacked_extension_path = 'C:/Users/charl/OneDrive/python/userprofile/profilename/Default/Extensions/pehaalcefcjfccdpbckoablngfkfgfgj/1.1_0'
chrome_options.add_argument('--load-extension={}'.format(unpacked_extension_path))
driver = webdriver.Chrome(options=chrome_options)
I also needed to add an extention to chrome while using selenium. What I did was first open the browser using selenium then add extention to the browser in the normal way like you would do in google chrome.
I've found the simpliest way ever.
So no ways were working for me, I tryed to make a crx file and more but nothing worked.
So I simply added the unpacked extension path like that:
options.add_argument('--load-extension={}'.format("HERE YOUR EXTENSION PATH"))
Replace the HERE YOUR EXTENSION PATH, by the path of your extension which is in one of your Chrome profiles.
To find it use this path on windows:
C:\Users[login_name]\AppData\Local\Google\Chrome\User Data\Default\Extensions
Then you have to choose the folder of the extension, it's by ID which you can find in the https://chrome.google.com/webstore/category/extensions, just type there your extension name and in the URL you'll get the ID.
Then there in this folder there will be the version of your extensions like: 10.0.3 choose it and it will be your path, so the path must end with the version.
Example:
options.add_argument('--load-extension={}'.format(r'C:\Users\nevo\AppData\Local\Google\Chrome\User Data\Default\Extensions\nkbihfbeogaeaoehlefnkodbefgpgknn\10.20.0_0'))
Note the "r" before the string to make it as raw, or I had to doble the backslashes.
And it works!