We Keep Coding

Python selenium - How to open regular chrome which we use for manual browsing using selenium

I am automating data scraping from the website https://www.macquarieinsights.com/. Now the website needs login only the first time. After that, if you open the website its already logged in. But, when I use selenium to visit the website, it asks for login every time I visit it via selenium. How can I use the regular chrome browser where it's already logged in, using selenium? Or is there any other alternative?

You can login that site to make the cookies saved in your local browser profile and after that you can use that browser profile.
Existing browser profile settings can be loaded from your PC storage as following:
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=C:\\Path") #Path to your chrome profile
driver = webdriver.Chrome(executable_path="C:\\Users\\chromedriver.exe",chrome_options=options)

Try to save cookies 🍪 made by the website on your login. And then provide that cookies to selenium while starting it...
Like :
Manually login to browser.
Use requests to get the webpage with the previous cookies
import requests
import browser_cookie3
cookies = browser_cookie3.chrome()
response = requests.get(url, cookies=cookies)
Don't forgot to install module using pip3 install browser-cookie3

How to open a browser tab after GET and POST requests in python

Im not sure if this is possible, but basically, I'm trying to open a tab in Chrome to show the outcome of my GET and POST requests. For instance, lets say im using Python requests to POST log in data to a site. I would want to then open a chrome tab after that POST request to see if I did it correctly, and also to continue on that webpage from there in Chrome itself. I know I can use response.text to check if my POST request succeeded, but is there a way I can physically open a chrome tab to see the result itself? Im guessing there would be a need to export some sort of cookies as well? Any help or insights would be appreciated.

When using selenium, you need to remove headless option for browser:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
#chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('<PATH_TO_CHROMEDRIVER>', options=chrome_options)
# load page via selenium
wd.get("<YOUR_TARGET_URL>")
# don't close browser
# wd.quit()
Also remove closing browser in the end of code.
After that browser will remain open and you will be able to continue work manually.

logging into a website and downloading a file

I would like to log into a website and download a file. I'm using selenium and the chromedriver. Would like to know if there is a better way. It currently opens up a chrome browser window and sends the info. I don't want to see the browser window opened up and the data being sent. Just want to send it and return the data into a variable.
from selenium import webdriver
driver = webdriver.Chrome()
def site_login(URL,ID_username,ID_password,ID_submit,name,pas):
driver.get(URL)
driver.find_element_by_id(ID_username).send_keys(name)
driver.find_element_by_id(ID_password).send_keys(pas)
driver.find_element_by_id(ID_submit).click()
URL = "www.mywebsite.com/login"
ID_username = "name"
ID_password = "password"
ID_submit = "submit"
name = "myemail#mail.com"
pas = "mypassword"
resp=site_login(URL,ID_username,ID_password,ID_submit,name,pas)

You can run chrome in headless mode. In which case, the chrome UI won't show up and still performing the task you were doing. Some article I found on this https://intoli.com/blog/running-selenium-with-headless-chrome/. Hope this helps.

First option: If you are able to change the driver, you can use phantom-js as driver. That was a headless browser and you can use it with selenium.
Second option: If the site are not dynamic (easily called it SPA) or you are able to trace packet (which can be done in chrome dev tools), you can directly use request with the help of beautifulsoup if you need to get some data on the page.

Just add this two lines
chrome_options = Options()
chrome_options.add_argument("--headless")
This should make chrome run in the background.

Python Selenium - URL won't open in browser instance

This particular URL won't open via python selenium script below. This same code works for most of the urls I have tried it on.
chrome_driver = 'chromedriver.exe' # Change this to your chrome driver path
driver = webdriver.Chrome(chrome_driver)
driver.get(url)
There is no any error message displayed, it will only show blank page and nothing will happen. When manually typed into regular chrome browser, it opens correctly but when manually into chrome browser instance, it won't work just like the script.
What is the solution to fix this?

Open URL using browser alongwith data

I need to open the browser using python.
I saw the Webbrowser module , which helps me fire a URL with a browser.
But i wanted to send a POST request by opening a browser.
The requests Module helps me to create a POST request alongwith the headers & Data .
Can i send the same by opening a browser tab or window ?

You should have a look at Selenium for Python
For example :
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.google.com")
assert "Google" in driver.title
driver.quit()
That will launch a Firefox browser and get the URL you passed to the driver. It's up to you to find out how you can perform what you want to do now ;)