We Keep Coding

Certain Aspects of a Website Are Being Blocked When Using Selenium

I have been trying to create a script to login and post an ad. However when directing selenium either the "post an ad" page or the "login" page, the URL doesn't load, and gives me a solid white screen. However when I direct the URL to the homepage or any other page, other than the ones already listed, the website loads just fine.
My problem solving:
I have read that my IP might have got blocked, however when I change my public IP address through my router, the problem persists.
I am currently unsure if the source of the problem stands from my IP or the possibly the website's scripts are only meant to detect and block scripts on certain pages.
I am also using chromedriver version 81.0.4044.69 and Chrome version Version 81.0.4044.129 (Official Build) (64-bit)
Here is my code:
from selenium import webdriver
browser = webdriver.Chrome()
browser.get('https://www.kijiji.ca/t-login.html?targetUrl=L3Atc2VsZWN0LWNhdGVnb3J5Lmh0bWw/XnIrUFRKMS9oU1cxc29PdXAxbjUveFE9PQ--')
The result, and no errors
Blank White Screen Found When Running The Program

Try faking the user-agent (some sites really are picky in regards to that) using this module
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
options = Options()
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options.add_argument(f'--user-agent={userAgent}')
driver = webdriver.Chrome(options=options)
driver.get("https://www.google.co.in")
driver.quit()

How to login and acquire cookie from browser - Python

Would it be possible to automate the process of logging into a site within the firefox browser, copy and store the cookie, place it in a DB table?
Scratching my head with this one.

A simple way to achieve this is through browser automation.
from selenium import webdriver
driver = webdriver.Firefox(executable_path='{/path/to/geckodriver}')
driver.get("http://google.com")
cookies = driver.get_cookies()
Note: Download geckodriver compatiable to your firefox.
Edit: Completly overlooked the fact that python's requests module has a cookies attribute. This would be quicker compared to browser automation.
import requests
resp = requests.get("http://google.com")
cookies = resp.cookies._cookies

Seleniumwire not logging all requests in chrome headless mode

I'm trying to capture all network logs using seleniumwire. When chromedriver is in normal mode, it is able to capture all requests. But when it is in headless mode, it is not capturing all requests.
I tried adding sleep(10), assert driver.last_request.response.status_code == 200
but neither helped.
Since seleniumwire is not that popular, I'm adding a sample guide below in the hope of getting people with knowledge of selenium to try a hand to help me fix the problem.
Working with seleniumwire
Installing seleniumwire
pip install seleniumwire
Sample script:
from seleniumwire import webdriver # Import from seleniumwire
# Create a new instance of the Chrome driver
driver = webdriver.Chrome()
# Go to the YouTube homepage.
driver.get('https://www.youtube.com')
# Access requests via the `requests` attribute
for request in driver.requests:
if request.response:
print(
request.path,
request.response.status_code,
request.response.headers['Content-Type']
)

try to capture all requests
options = {
'ignore_http_methods': [] # Capture all requests, including OPTIONS requests
}
driver = webdriver.Chrome("C:\chromedriver.exe",seleniumwire_options=options)
In default it ignores OPTIONS method

When chrome browser is opened by selenium, it uses it's own profile rather than the default one present. Try using custom profile, for chrome you can use ChromeOptions class use a custom profile and try.

Selenium web driver Firefox opening blank page

I'd like to ask something about Selenium library in Python.
I'm trying to open a webpage, directly log onto it, and access another webpage behind it (I wanted to navigate on the website after the login) with a Python script. I've found the following code on the Internet but I have a problem with the line:
browser = webdriver.Firefox()
It just opens a blank page in Firefox and it looks like the script get stuck with it and does nothing afterwards. I tried in the Python interpreter and it's the same, it opens a blank page in Firefox and I lose the hand (I can't enter other commands).
python interpreter blocked:
I'm using Selenium-3.3.1 and I work under CentOS 6.5.
Is it normal? Am I missing something obvious?
Here is my code:
#!usr/bash/python
from selenium import webdriver
from selenium.webdriver.support import ui
from selenium.webdriver.common.keys import Keys
def loadedPage(browser):
return browser.find_element_by_tag_name("body") != None
browser = webdriver.Firefox() #supposedly just a firefox webdrive instance creation
browser.get("http://machine/machineDir/index.php")
wait = ui.WebDriverWait(browser, 10)
wait.until(loadedPage)
username=browser.find_element_by_id("username")
username.send_keys("userTest")
passwd=browser.find_element_by_id("password")
passwd.send_keys("userTestpass")
passwd.send_keys(Keys.RETURN)

As you are using selenium 3, firefox browser can't be instantiate directly, you need to configure gecko driver for the same.
System.setProperty("webdriver.gecko.driver","path of geckodriver.exe");

I fixed it using the right version of Selenium for my old Firefox.
Firefox version: 17.0.10
Selenium version installed: 2.40

Open URL using browser alongwith data

I need to open the browser using python.
I saw the Webbrowser module , which helps me fire a URL with a browser.
But i wanted to send a POST request by opening a browser.
The requests Module helps me to create a POST request alongwith the headers & Data .
Can i send the same by opening a browser tab or window ?

You should have a look at Selenium for Python
For example :
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.google.com")
assert "Google" in driver.title
driver.quit()
That will launch a Firefox browser and get the URL you passed to the driver. It's up to you to find out how you can perform what you want to do now ;)