This is the code that I'm using, I've tried numerous other solutions from stackoverflow, but everytime I make a request to httpbin.org/ip I see my own IP.
# Selenium Logging
selenium_logger = logging.getLogger('seleniumwire')
selenium_logger.setLevel(logging.ERROR)
# Web driver settings
# Profile
profile = webdriver.FirefoxProfile()
profile.accept_untrusted_certs = True
# Capabilities
firefox_capabilities = webdriver.DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
# Options
options = Options()
options.headless = False
brightdata_proxy = "personal_brighdata_url"
firefox_proxies = Proxy()
firefox_proxies.ssl_proxy = brightdata_proxy
firefox_proxies.add_to_capabilities(firefox_capabilities)
self._driver = webdriver.Firefox(executable_path=GeckoDriverManager().install(),
firefox_profile=profile,
capabilities=firefox_capabilities,
options=options,
)
Can anybody help me to figure those proxies out ?
Related
I am trying to make a proxy in python using selenium i have a list of free proxies from https://free-proxy-list.net/ but the ip i get is always the same, there is something wrong with this setup?
PROXY = proxy_list[11]
options = Options()
options.headless = False
webdriver.DesiredCapabilities.FIREFOX['proxy'] = {
"httpProxy":PROXY,
"ftpProxy":PROXY,
"sslProxy":PROXY,
"noProxy":None,
"proxyType":"MANUAL",
"class":"org.openqa.selenium.Proxy",
"autodetect":False
}
driver = webdriver.Firefox(options=options, executable_path=r'geckodriver.exe')
# Set the interceptor on the driver
driver.request_interceptor = interceptor
driver.get(url)
time.sleep(5)
driver.quit()
EDIT: i am running according to the response the following code:
url = 'https://www.whatismyip.com/es/'
PROXY = proxy_list[6]
options = Options()
options.headless = False
firefox_capabilities = webdriver.DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
firefox_capabilities['proxy'] = {
'proxyType': "MANUAL",
'httpProxy': PROXY,
'ftpProxy': PROXY,
'sslProxy': PROXY
}
driver = webdriver.Firefox(options=options, executable_path=r'geckodriver.exe', capabilities = firefox_capabilities)
# Set the interceptor on the driver
driver.request_interceptor = interceptor
driver.get(url)
time.sleep(50)
driver.quit()
But the ip in the webpage is always the same, dunno if missunderstood the use of the proxy or i am doing something wrong
If you use Firefox browser,
...
# set proxy
firefox_capabilities = webdriver.DesiredCapabilities.FIREFOX
firefox_capabilities['marionette'] = True
firefox_capabilities['proxy'] = {
'proxyType': "MANUAL",
'httpProxy': PROXY,
'ftpProxy': PROXY,
'sslProxy': PROXY
}
driver = webdriver.Firefox(options=options, executable_path=r'geckodriver.exe', capabilities = firefox_capabilities)
If you use Chrome,
...
chrome_options = Options()
chrome_options.add_argument('--proxy-server=' + PROXY)
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path = 'chromedriver.exe', options=chrome_options)
Hey there I am trying to automate a buying process but there is some abnormal behaviour, that sometimes (very few) the bot is able to make till the end and the rest of the time it is being detected in the middle or in the beggining of the process, I have heard that the IP I am using rotates in the middle of the process and this can be the reason, I am attaching the code for selenium initiazation I am doing, I have used every possible thing, do help please.
def __init__(self,url):
self.url = url
def get_selenium_res(self):
try:
software_names = [SoftwareName.CHROME.value]
operating_systems = [OperatingSystem.WINDOWS.value,OperatingSystem.LINUX.value]
user_agent_rotator = UserAgent(software_names=software_names,operating_systems=operating_systems,limit=1000)
user_agent = user_agent_rotator.get_random_user_agent()
print(user_agent)
options = Options()
options.add_argument('--no-sandbox')
options.add_argument("start-maximized")
options.add_argument('--disable-gpu')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument(f'user-agent={user_agent}')
PROXY = "http://pl.smartproxy.com:20000"
proxy = Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.autodetect = False
capabilities = webdriver.DesiredCapabilities.CHROME
proxy.http_proxy = PROXY
proxy.ssl_proxy = PROXY
proxy.add_to_capabilities(capabilities)
browser = uc.Chrome(executable_path='chromedriver.exe',chrome_options=options)#Remove navigator.webdriver Flag using JavaScript
browser.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
I'm unable lo load my Firefox profile on the Selenium Webdriver.
i have tried several ways commented here on SO, but non did the trick. Either i loose connection to the driver or the profile does not get loaded.
this is what i have so far:
from selenium import webdriver
options = Options()
options.headless = False
fp = webdriver.FirefoxProfile(r'C:\\Users\\xxXX\\AppData\\Local\\Mozilla\\Firefox\\Profiles\\y73h6ogb.default-release')
driver = webdriver.Firefox(fp, options=options, executable_path='geckodriver/geckodriver.exe')
driver.get(URL)
According to the documentation you can load a profile like this:
from selenium.webdriver import Firefox, FirefoxOptions, FirefoxProfile
options = FirefoxOptions()
options.headless = True
profile = FirefoxProfile("PATH_TO_PROFILE\\1234asdf.some_profile")
driver = Firefox(
firefox_profile=profile,
executable_path="PATH_TO_DRIVER\\geckodriver.exe",
options=options,
)
driver.get("https://python.org")
driver.close()
Or by adding the profile to the options:
...
options = FirefoxOptions()
options.headless = True
options.profile = FirefoxProfile("PATH_TO_PROFILE\\1234asdf.some_profile")
driver = Firefox(
executable_path="PATH_TO_DRIVER\\geckodriver.exe",
options=options,
)
...
By using webdriver for firefox, You will be able to do that!
I use the brave browser as my default browser but I was wondering if it is possible to open up my regular browser profile when I run my code instead of the new instance where it does not have all my bookmarks and passwords?
Thanks in advance!
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time
import datetime
day = datetime.datetime.now()
dag = day.weekday()
def Testing():
if dag >= 5:
return Weekend()
else:
return Weekday()
def Weekend():
options = Options()
options.add_argument("--window-size=1920,1080")
## options.add_argument("/Users/vadim/Library/Application Support/BraveSoftware/Brave-Browser")
options.binary_location = '/Applications/Brave Browser.app/Contents/MacOS/Brave Browser'
driver_path = '/usr/local/bin/chromedriver'
driver = webdriver.Chrome(options=options, executable_path=driver_path)
driver.get('https://outlook.live.com/mail/0/inbox')
Outlook_Aanmelden = driver.find_element_by_xpath('/html/body/header/div/aside/div/nav/ul/li[2]/a')
Outlook_Aanmelden.click()
Email_Field = driver.find_element_by_xpath('//*[#id="i0116"]')
Email_Field.send_keys('#live.com')
Outlook_Volgende = driver.find_element_by_xpath('//*[#id="idSIButton9"]')
Outlook_Volgende.click()
time.sleep(0.5)
Password_Field = driver.find_element_by_xpath('//*[#id="i0118"]')
Password_Field.send_keys('pass')
Password_Field.send_keys(Keys.ENTER)
Inlog_Outlook = driver.find_element_by_xpath('//*[#id="idSIButton9"]')
Inlog_Outlook.click()
driver.execute_script("window.open('https://youtube.com');")
def Weekday():
options = Options()
options.add_argument("--window-size=1920,1080")
## options.add_argument("/Users/vadim/Library/Application Support/BraveSoftware/Brave-Browser")
options.binary_location = '/Applications/Brave Browser.app/Contents/MacOS/Brave Browser'
driver_path = '/usr/local/bin/chromedriver'
driver = webdriver.Chrome(options=options, executable_path=driver_path)
driver.get('https://outlook.live.com/mail/0/inbox')
Outlook_Aanmelden = driver.find_element_by_xpath('/html/body/header/div/aside/div/nav/ul/li[2]/a')
Outlook_Aanmelden.click()
Email_Field = driver.find_element_by_xpath('//*[#id="i0116"]')
Email_Field.send_keys('#live.com')
Outlook_Volgende = driver.find_element_by_xpath('//*[#id="idSIButton9"]')
Outlook_Volgende.click()
time.sleep(0.5)
Password_Field = driver.find_element_by_xpath('//*[#id="i0118"]')
Password_Field.send_keys('pass')
Password_Field.send_keys(Keys.ENTER)
Inlog_Outlook = driver.find_element_by_xpath('//*[#id="idSIButton9"]')
Inlog_Outlook.click()
driver.execute_script("window.open('https://youtube.com');")
Testing()
So yeah been having this problem for some time now and haven't found anyone who seems to know the solution just yet, maybe you can help me out?
You need to load your chrome with the user profile, like so. Assuming it is chrome and not brave.
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('user-data-dir=<path to chrome profile>')
browser = webdriver.Chrome(chrome_options=chrome_options)
The location for Chrome’s default profile folder differs depending on your platform. The locations are:
Windows 7, 8.1, and 10: C:\Users\<username>\AppData\Local\Google\Chrome\User Data\Default
Mac OS X El Capitan: Users/<username>/Library/Application Support/Google/Chrome/Default
Linux: /home/<username>/.config/google-chrome/default
Hi I'm using selenium in my web app on heroku. I used it to login to instagram but it raises a BadStatusLine error whenever selenium is sending keys to the user field/password field.
My code:
display = Display(visible=0, size=(800, 600))
display.start()
options = Options()
options.binary_location = "/app/.apt/usr/bin/google-chrome-stable"
driver = webdriver.Chrome(chrome_options = options)
driver.get("http://www.instagram.com/")
#click login button to show login form
driver.implicitly_wait(20) # seconds
login_btn = driver.find_element_by_link_text("Log in")
login_btn.click()
#enter username and password
driver.implicitly_wait(20) # seconds
driver.find_element_by_name("username").send_keys(login)
driver.find_element_by_name("password").send_keys(password)
Error:
For anyone who need, you will need
-Two buildpacks
-Set stack to cedar-14
-Have chromedriver in your files. Point to it and set chromedriver options
from selenium.webdriver.chrome.options import Options
dir_path = os.path.dirname(os.path.realpath(__file__))
chromedriver_path = os.path.join(dir_path, "chromedriver")
options = Options()
options.binary_location = "/app/.apt/usr/bin/google-chrome-stable"
driver = webdriver.Chrome(executable_path = chromedriver_path, chrome_options = options)