We Keep Coding

selenium (in python) don't complete the page loading

I'm trying to scrap this site: https://www.politicos.org.br/Ranking
and my cell on jupyter notebook don't complete the loading.
The page has a Cookies button to accept, but I can't figure out how I can click on it. And I don't know if this is the problem.
import re
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from time import sleep
options = Options()
options.add_argument('window-size=1000,800')
navegador = webdriver.Chrome(options=options)
navegador.get('https://www.politicos.org.br/Ranking')
sleep(3)
click_dep = navegador.find_element(By.XPATH, '//*[#id="__next"]/div[2]/div[1]/div[4]/button')
click_dep.click()
sleep(1)
I'm using python 3 on jupyter notebook,
thanks for you attention.

I found the HTML code for Cookies button : <button class="mb-3">Aceitar</button>
So you can easily write the code to click button.
navegador.find_element(By.XPATH, '//*[#class="mb-3"]').click()

I can solve this using this on my Options:
options.page_load_strategy = 'eager'
from another post on stackoverflow: "Eager" page loading strategy will make WebDriver wait until the initial HTML document has been completely loaded and parsed, and discards loading of stylesheets, images and subframes (DOMContentLoaded event fire is returned). stackoverflow.com/questions/66358904/… And, I use the zerg468 sugestion to click on that button

Selenium get() redirects to another url

I'm trying to navigate to the following page and extract the html https://www.automobile.it/annunci?b=data&d=DESC, but everytime i call the get() method it looks like the website redirects me to the another page, always the same one which is https://www.automobile.it/torrenova?radius=100&b=data&d=DESC.
here's the simple code i'm running:
from selenium import webdriver
driver = webdriver.Chrome(executable_path=ex_path)
driver.get("https://www.automobile.it/annunci?b=data&d=DESC")
html=driver.page_source
if i do the same thing using the request module i don't get redirected
import requests
html=requests.get("https://www.automobile.it/annunci?b=data&d=DESC")
i don't understand why it's behaving like this, any ideas?

Use driver.delete_all_cookies()
from selenium import webdriver
driver = webdriver.Chrome(executable_path=ex_path)
driver.delete_all_cookies()
driver.get("https://www.automobile.it/annunci?b=data&d=DESC")
html=driver.page_source
PS: be also warned: Page_source will not get you the completed DOM as rendered.

Well you can clear browser cache by using the below code :
I am assuming that you are using chrome.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(executable_path=ex_path)
driver.get('chrome://settings/clearBrowserData')
driver.find_element_by_xpath('//settings-ui').send_keys(Keys.ENTER)
driver.get("https://www.automobile.it/annunci?b=data&d=DESC")

Give upload file path to Instagram with Selenium and python

I'm testing some web scraping on Instagram with Selenium and Python.
In this case I want to upload a picture.
Normally you have to click on the upload icon and choose the file from a window. How can I manage it with Selenium?
I tried:
driver.find_element_by_class_name("coreSpriteFeedCreation").send_keys('C:\\path-to-file\\file.jpg')
and also with find_element_by_xpath but I get an exception:
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot focus element
I tried also only with click() but nothing happens.
Any Idea?
EDIT
Thanks to #homersimpson comment I tried this:
actions = ActionChains(driver)
element = driver.find_element_by_class_name("coreSpriteFeedCreation")
actions.move_to_element(element)
actions.click()
actions.send_keys('C:\\path-to-file\\file.jpg')
actions.perform()
Now the window to choose the file appears. The problem is that I would like to avoid this window and give directly the path of my file.

If right understand, you are trying to avoid handling with a native window. You can try this:
# get all inputs
inputs = driver.find_elements_by_xpath("//input[#accept = 'image/jpeg']").send_keys(os.getcwd() + "/image.png")
Now you can try all of them. I don't know which of them will work.
More about os.getcwd() is here
To be able to perform this code you have to have an element like this:
<input type="file" name="fileToUpload" id="fileToUpload2" class="fileToUpload">
EDIT:
It looks like instagram turned of input fields interaction for posts. For Account image it still works, but not for posting. I assume it is was done to prevent bots to post images. Anyway, there is a solution for this problem. You can use AutoIt like this:
import autoit
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
ActionChains(driver).move_to_element( driver.find_element_by_xpath("//path/to/upload/button")).click().perform()
handle = "[CLASS:#32770; TITLE:Open]"
autoit.win_wait(handle, 60)
autoit.control_set_text(handle, "Edit1", "\\file\\path")
autoit.control_click(handle, "Button1")

I think I may have found a solution that works for me. I found if you first have the bot click the plus icon while the browser is in the mobile view.
self.driver.find_element_by_xpath("/html/body/div[1]/section/nav[2]/div/div/div[2]/div/div/div[3]")\
.click()
after that, I would immediately send my file to an input tag in the HTML and I find you may need to play around as to which one works but I find the last input tag worked for me.
self.driver.find_element_by_xpath("/html/body/div[1]/section/nav[2]/div/div/form/input")\
.send_keys("/image/path.jpg")
The one weird thing about this is you will have a popup menu on top of the page but your code will still function with this window displayed over the window you are working on.

Addition to HumbleFox's answer. To solve his problem regarding the pop-up box not closing or the file pop-up box not closing (bug)
The solution to this is to make the browser headless here's a part of my code for example:
mobile_emulation = { "deviceName": "Pixel 2" }
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_experimental_option("mobileEmulation", mobile_emulation)
chrome_options.binary_location = self.opt.binary_location
self.driver = webdriver.Chrome(executable_path=self.path, options=chrome_options)

Element not visible with pyvirtualdisplay and selenium in python

I'm trying a simple browser operation where I locate a username element on a website and then try to login. I'm using selenium and python to do this. Here's some simple code that works on my own local machine. The code opens a browser on my computer and then navigates to the correct username box and enters the username.
from selenium import webdriver
browser = webdriver.Firefox()
browser.get(my_url)
username_element = browser.find_elements_by_name("USERNAME")[0]
username_element.clear()
username_element.send_keys(my_username)
However, when I try to deploy the same code on an AWS server using pyvirtualdisplay so that Firefox doesn't need to pop up, it no longer works.
from pyvirtualdisplay import Display
from selenium import webdriver
display = Display(visible=0, size=(800, 600))
display.start()
browser = webdriver.Firefox()
browser.get(my_url)
username_element = browser.find_elements_by_name("USERNAME")[0]
username_element.clear()
username_element.send_keys(my_username)
The element is definitely found, but I get the element not visible error:
selenium.common.exceptions.ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with
which is confirmed by:
>> username_element.is_displayed()
False
I've tried various things I found on SO including:
making sure xvfb and xephyr are installed
adding a browser.implicitly_wait(30)
trying a WebDriverWait(browser,30).until(EC.visibility_of_element_located((By.NAME, "USERNAME"))) which times out
Any ideas on how to solve this?

You can scroll screen:
browser.execute_script("window.scrollTo(0, 600)")

Figured it out after taking a screenshot. Turns out my screen display wasn't set large enough. Changing the display size to (1600,900) solved the problem.

How to use Selenium with Python?

How do I set up Selenium to work with Python? I just want to write/export scripts in Python, and then run them. Are there any resources for that? I tried googling, but the stuff I found was either referring to an outdated version of Selenium (RC), or an outdated version of Python.

You mean Selenium WebDriver?
Huh....
Prerequisite: Install Python based on your OS
Install with following command
pip install -U selenium
And use this module in your code
from selenium import webdriver
You can also use many of the following as required
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
Here is an updated answer
I would recommend you to run script without IDE... Here is my approach
USE IDE to find xpath of object / element
And use find_element_by_xpath().click()
An example below shows login page automation
#ScriptName : Login.py
#---------------------
from selenium import webdriver
#Following are optional required
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
baseurl = "http://www.mywebsite.com/login.php"
username = "admin"
password = "admin"
xpaths = { 'usernameTxtBox' : "//input[#name='username']",
'passwordTxtBox' : "//input[#name='password']",
'submitButton' : "//input[#name='login']"
}
mydriver = webdriver.Firefox()
mydriver.get(baseurl)
mydriver.maximize_window()
#Clear Username TextBox if already allowed "Remember Me"
mydriver.find_element_by_xpath(xpaths['usernameTxtBox']).clear()
#Write Username in Username TextBox
mydriver.find_element_by_xpath(xpaths['usernameTxtBox']).send_keys(username)
#Clear Password TextBox if already allowed "Remember Me"
mydriver.find_element_by_xpath(xpaths['passwordTxtBox']).clear()
#Write Password in password TextBox
mydriver.find_element_by_xpath(xpaths['passwordTxtBox']).send_keys(password)
#Click Login button
mydriver.find_element_by_xpath(xpaths['submitButton']).click()
There is an another way that you can find xpath of any object -
Install Firebug and Firepath addons in firefox
Open URL in Firefox
Press F12 to open Firepath developer instance
Select Firepath in below browser pane and chose select by "xpath"
Move cursor of the mouse to element on webpage
in the xpath textbox you will get xpath of an object/element.
Copy Paste xpath to the script.
Run script -
python Login.py
You can also use a CSS selector instead of xpath. CSS selectors are slightly faster than xpath in most cases, and are usually preferred over xpath (if there isn't an ID attribute on the elements you're interacting with).
Firepath can also capture the object's locator as a CSS selector if you move your cursor to the object. You'll have to update your code to use the equivalent find by CSS selector method instead -
find_element_by_css_selector(css_selector)

There are a lot of sources for selenium - here is good one for simple use Selenium, and here is a example snippet too Selenium Examples
You can find a lot of good sources to use selenium, it's not too hard to get it set up and start using it.

You just need to get selenium package imported, that you can do from command prompt using the command
pip install selenium
When you have to use it in any IDE just import this package, no other documentation required to be imported
For Eg :
import selenium
print(selenium.__filepath__)
This is just a general command you may use in starting to check the filepath of selenium