I have a scenario in which I need to process some the web request automatically and finally show the result in the browser. The website I am querying uses partial response and ajax call to get the whole page.
Is there a way in which I can open a web browser with response object or is there a way in which I can pass data and cookies with the url to web browser.
You can use selenium for it
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://www.your-website.com?q=test_param")
# You can sendout a wait to view the results
driver.close()
You can very well use chrome driver too if you are very much concerned about the browser
Related
I'm using Python with Selenium to extract some data from a webpage through automation. That webpage normally requires one-time yubikey based authentication per day.
However, when .get()ing that page using Chrome Webdriver, it asks for authentication each time and I need to add wait() in my script in order to manually authenticate first before reaching the target page.
I think the webdriver controlled browser doesn't save authentication sessions like a normal browser does.
Is it possible to make the webdriver controlled browser save authentication sessions normally so that I don't have to manually authenticate every time I get() it?
This example will use the cookies to log you into stackoverflow.com after you login and save the cookies.
from selenium import webdriver
import time
import pickle
driver = webdriver.Firefox(executable_path=r'C:\\Path\\To\\geckodriver.exe')
driver.get('https://stackoverflow.com/')
time.sleep(60)
#########################
#LOGIN TO STACKOVERFLOW
########################
pickle.dump(driver.get_cookies(), open("C:/Path/to/cookie/cookies.pkl","wb"))
driver.close()
time.sleep(5)
driver = webdriver.Firefox(executable_path=r'C:\\Path\\To\\geckodriver.exe')
driver.get("https://stackoverflow.com/")
cookies = pickle.load(open("C:/Path/to/cookie/cookies.pkl", "rb"))
for cookie in cookies:
driver.add_cookie(cookie)
driver.get("https://stackoverflow.com/")
After the cookie is saved you can just use the cookie.
from selenium import webdriver
import time
import pickle
driver = webdriver.Firefox(executable_path=r'C:\\Path\\To\\geckodriver.exe')
driver.get("https://stackoverflow.com/")
cookies = pickle.load(open("C:/Path/to/cookie/cookies.pkl", "rb"))
for cookie in cookies:
driver.add_cookie(cookie)
driver.get("https://stackoverflow.com/")
Would it be possible to automate the process of logging into a site within the firefox browser, copy and store the cookie, place it in a DB table?
Scratching my head with this one.
A simple way to achieve this is through browser automation.
from selenium import webdriver
driver = webdriver.Firefox(executable_path='{/path/to/geckodriver}')
driver.get("http://google.com")
cookies = driver.get_cookies()
Note: Download geckodriver compatiable to your firefox.
Edit: Completly overlooked the fact that python's requests module has a cookies attribute. This would be quicker compared to browser automation.
import requests
resp = requests.get("http://google.com")
cookies = resp.cookies._cookies
Is there a way to open a browser from a Python application, while posting data to it in POST method?
I tried using webbrowser.open(url) to open the browser window, but I need to also post some data to it (via POST variables), to make sure the page opens with the proper information.
As an example, I would like to open http://duckduckgo.com, while posting "q=mysearchterm" as POST data to it, so the page will open with pre-filled data.
I believe you should be using a Web Driver such as Selenium. Here is an example:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser = webdriver.Chrome(r"PATHTO\chromedriver.exe") # download the chromedriver and provide the full path here
browser.get('http://duckduckgo.com/')
search = browser.find_element_by_name('q')
search.send_keys("mysearchterm")
search.send_keys(Keys.RETURN)
time.sleep(15)
I'm using selenium to pull some automated phone reporting from our phone system (Barracuda Cudatel, very nice small business system but it doesn't have an API for what I need). There is a button on the report page that has some javascript attached to it on a click listener that then tells the browser to download the file.
Obviously selenium isn't really designed to pull files like this, however all I'm trying to do is get the href of the url that would have been sent to the browser. I can then turn around and use the session credentials with requests to pull the file and do processing on it.
How do I do the following (In Python):
Query for the event listener for 'click'
Fire off the javascript
Get the resulting URL
Edit: I'm aware download location can be configured on the browser in selenium however I'm not interested in completing this task in that fashion. This is running against a selenium grid of 20 machines and the request could be routed to any of them. Since I can't pull the file through selenium I'm going to just pull it directly with requests.
Code I'm twiddling with is below.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from time import sleep
dcap = webdriver.DesiredCapabilities.CHROME
driver = webdriver.Remote(command_executor='http://gridurl:4444/wd/hub', desired_capabilities=dcap)
driver.get("http://cudatelurl")
driver.find_element_by_name("__auth_user").send_keys("user")
driver.find_element_by_name("__auth_pass").send_keys("password")
driver.find_element_by_id("manage").click()
driver.get("http://cudatelurl/#!/cudatel/cdrs")
sleep(5)
date_dropdown = Select(driver.find_element_by_xpath('//*[#id="cui-content-inner"]/div[3]/div/div/div/div[2]/div/div/div[1]/div[2]/div/select'))
date_dropdown.select_by_value("last_week")
# This is the element that has javascript attached to it the click register is
# button.widgetType.csvButtonWidget.widgetized.actionButtonType.table-action
# but I'd prefer to not hard code it
driver.find_element_by_xpath('//*[#id="cui-content-inner"]/div[3]/div/div/div/div[2]/div/div/div[1]/div[2]/button[1]')
print(driver.get_cookies())
print(driver.title)
sleep(10)
driver.close()
driver.quit()
You can still approach it with selenium by configuring the target directory for the file of a specific mime-type to be automatically downloaded (without the Save As dialog), see:
Download PDF files automatically in Firefox using Selenium WebDriver
Access to file download dialog in Firefox
Firefox + Selenium WebDriver and download a csv file automatically
I need to open the browser using python.
I saw the Webbrowser module , which helps me fire a URL with a browser.
But i wanted to send a POST request by opening a browser.
The requests Module helps me to create a POST request alongwith the headers & Data .
Can i send the same by opening a browser tab or window ?
You should have a look at Selenium for Python
For example :
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.google.com")
assert "Google" in driver.title
driver.quit()
That will launch a Firefox browser and get the URL you passed to the driver. It's up to you to find out how you can perform what you want to do now ;)