We Keep Coding

How to prevent Instagram temporary requests block in selenium?

Iam trying to write my own python script to find an account top followed followers, and it seems to work fine, however after a while or after running the script more than 1-2 times, instagram gives me a try again error, which ive searched and found its Instagram temporarily blocking my ip as i have given to many requests at once.
Does anyone know a way to get around this?
MY CODE
"""
WHAT DOES THIS SCRIPT ACTUALLY DO?:
This script enables you to scrape all your followers and then find X top followed followers.
--------------------------------------------------------------------------------------------
NOTICE:
Unfortunately it is very hard now a days to scrape social media sites, due to
timeout issues, to many pings in a set time and other request restrictions.
So this script can only be ran 1-3 times a day.
I've tried also using exciting API's but all these are either too slow, or simply
show a '428' to many requests error.
"""
import instaloader
from selenium import webdriver
import time
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.keys import Keys
from rich.console import Console
from rich.table import Column, Table
# Global vars
L = instaloader.Instaloader()
URL = "https://www.instagram.com/{}/"
usernameGlobal = None
passwordGlobal = None
console = Console()
def get_followers():
# Login
while True: # Keep running if password/username was wrong
try:
global usernameGlobal, passwordGlobal
print("\n"+"*-=-*"*5)
usernameGlobal = input("> Enter your username: ")
passwordGlobal = input("> Enter your password: ")
L.login(usernameGlobal, passwordGlobal)
print("\n"+"-"*28+"\n> Successfully Logged In!")
print("> Please leave this program running in the background")
print("> Until you see the 'FINISHED' message'"+"\n"+"-"*28)
break
except:
print("\n"+"-"*28+"\n> Wrong Username / Password"+"\n"+"-"*28)
# Obtain profile metadata
profile = instaloader.Profile.from_username(L.context, usernameGlobal)
follow_list = []
# Loop through each follower and add to list
for followee in profile.get_followers():
follow_list.append(followee.username)
return follow_list
def scrape_data(username):
driver.get(URL.format(username))
FOLLOWERS = 0
try:
try:
FOLLOWERS = driver.find_element_by_xpath('/html/body/div[1]/section/main/div/header/section/ul/li[2]/a/span').text
except: # For people who you don't follow but follow you and have private accounts
FOLLOWERS = driver.find_element_by_xpath('/html/body/div[1]/section/main/div/header/section/ul/li[2]/span/span').text
except:
print("\n"+"-"*28+"\n> Please try this script again later!"+"\n"+"-"*28)
result = ''.join([i for i in FOLLOWERS if i.isdigit()])
return int(float(result))
def driver_login():
driver.get("https://www.instagram.com")
time.sleep(3)
element = driver.find_element_by_xpath("//input[#name='username']")
element.send_keys(usernameGlobal)
element = driver.find_element_by_xpath("//input[#name='password']")
element.send_keys(passwordGlobal)
element.send_keys(Keys.RETURN)
time.sleep(3)
# -- This is for if you have two factor authentication enabled --
# element = driver.find_element_by_xpath("//input[#name='verificationCode']")
# key = input("Enter Activation key: ")
# element.send_keys(key)
# element.send_keys(Keys.RETURN)
# time.sleep(3)
def output_result(size, result):
n_input = 0
# Get user to select how many of the top followed followers they want
while True:
try:
print("\n"+"*-=-*"*10)
n_input = int(input("> How many of your top followed followers do you want to see?\n> E.g 5 for top 5.\n> "))
if n_input > size:
continue
break
except:
print("\n"+"-"*28+"\n> Invalid input. (Must be a number & less then your follower count)"+"\n"+"-"*28)
# Make the table for a clean user friendly output and print it out
table = Table(show_header=True, header_style="bold magenta")
table.add_column("Your Followers", style="dim", width=12)
table.add_column("There Follower Count")
for x in range(n_input):
table.add_row(
list(result.keys())[x-1],
list(result.values())[x-1]
)
console.print(table)
return
if __name__ == "__main__":
list_of_followers = get_followers()
# Initialize the selenium driver
driver = webdriver.Chrome(ChromeDriverManager().install())
driver_login()
result = {}
for follower in list_of_followers:
followers = scrape_data(follower)
result[follower] = followers
# Sort the dictionary by descending order
result = dict(sorted(result.items(), key=lambda x: x[1], reverse=True))
print("\n> FINISHED")
driver.quit()
output_result(len(list_of_followers), result)
exit(0)

You can potentially make unlimited requests if you use proxies. You can buy thousands of proxies from various sites and rotate them in a dictionary.
Simply add a list of proxies to your GET request and enjoy:
proxyDict = {
"http" : http_proxy,
"https" : https_proxy,
"ftp" : ftp_proxy
}
r = requests.get(url, headers=headers, proxies=proxyDict)
Also for Selenium, from this answer:
PROXY = "1.111.111.1:8080" #your proxy
chrome_options = WebDriverWait.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % PROXY)
chrome = webdriver.Chrome(chrome_options=chrome_options)
chrome.get("instagram.com")

Run python class only once and save output for later use

So I have this craigslist scraper project I am working on and I am running into a potential problem. I have this file url.py that has a UrlObj class and getters and setters. In my main.py, I am instantiating that object and getting the completed url back to be sent to the Job class in my main.py to do its scraping stuff.
I would like to deploy this in the cloud in the future and have it run on a time interval (i.e. every day, 4 hours, etc), but I have noticed a problem. Every time this program is ran, the UrlObj class will be called, prompting the user to enter the relevant data to construct the URL. Since this will be in the cloud running in the background, no one will be able to input the prompts every time its built and ran.
What I want is for url.py and UrlObj to be called only once, in the beginning to allow the user to input and populate the necessary fields to construct the url. Then, every time the program is built and ran, the url the user made in the beginning should be used, not calling url.py and UrlObj to prompt the user again to type in inputs since it will be running in the cloud and on a time interval.
Is it too naive to think to set conditions around url = UrlObj().url to make sure it runs once. Like an if statement or while loop?
url.py:
class UrlObj:
def __init__(self):
self.location = self.get_location() # Location(i.e. City) being searched
self.postal_code = self.get_postal_code() # Postal code of location being searched
self.query = self.get_query() # Search for the type of items that will be searched
self.max_price = self.get_max_price() # Max price of the items that will be searched
self.radius = self.get_radius() # Radius of the area searched derived from the postal code given previously
self.url = f"https://{self.location}.craigslist.org/search/sss?&max_price={self.max_price}&postal={self.postal_code}&query={self.query}&20card&search_distance={self.radius}"
def get_location(self):
location = input("Please enter the location: ")
return location
def get_postal_code(self):
postal_code = input("Please enter the postal code: ")
return postal_code
def get_query(self):
query = input("Please enter the item: ")
return query
def get_max_price(self):
max_price = input("Please enter the max price: ")
return max_price
def get_radius(self):
radius = input("Please enter the radius: ")
return radius
main.py:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import pandas as pd
from url import *
class Job():
def __init__(self):
self.driver = webdriver.Chrome(r"C:\Program Files\chromedriver") # Path of Chrome web driver
self.delay = 5 # The delay the driver gives when loading the web page
# Load up the web page
# Gets all relevant data on the page
# Goes to next page until we are at the last page
def load_craigslist_url(self, url):
data = []
self.driver.get(url)
while True:
try:
wait = WebDriverWait(self.driver, self.delay)
wait.until(EC.presence_of_element_located((By.ID, "searchform")))
data.append(self.extract_post_titles())
WebDriverWait(self.driver, 2).until(
EC.element_to_be_clickable((By.XPATH, '//*[#id="searchform"]/div[3]/div[3]/span[2]/a[3]'))).click()
except:
break
return data
# # Extracts all relevant information from the web-page and returns them as individual lists
def extract_post_titles(self):
all_posts = self.driver.find_elements_by_class_name("result-row")
dates_list = []
titles_list = []
prices_list = []
distance_list = []
for post in all_posts:
title = post.text.split("$")
if title[0] == '':
title = title[1]
else:
title = title[0]
title = title.split("\n")
price = title[0]
title = title[-1]
title = title.split(" ")
month = title[0]
day = title[1]
title = ' '.join(title[2:])
date = month + " " + day
if not price[:1].isdigit():
price = "0"
int(price)
raw_distance = post.find_element_by_class_name(
'maptag').text
distance = raw_distance[:-2]
titles_list.append(title)
prices_list.append(price)
dates_list.append(date)
distance_list.append(distance)
return titles_list, prices_list, dates_list, distance_list
# # Kills browser
def kill(self):
self.driver.close()
#staticmethod
def organizeResults(results):
titles_list = results[0][0]
prices_list = list(map(int, results[0][1]))
dates_list = results[0][2]
distance_list = list(map(float, results[0][3]))
list_of_attributes = []
for i in range(len(titles_list)):
content = {'Listing': titles_list[i], 'Price': prices_list[i], 'Date posted': dates_list[i],
'Distance from zip': distance_list[i]}
list_of_attributes.append(content)
list_of_attributes.sort(key=lambda x: x['Distance from zip'])
return list_of_attributes
#staticmethod
def to_csv(dictionary):
df = pd.DataFrame(dictionary)
df.to_csv('data.csv', index=False)
if __name__ == '__main__':
# This should be called only once!!!
# Then the 'url' should be used every time main.py is built and ran, and not be constructed again by calling 'UrlObj().url'
url = UrlObj().url
scraper = Job()
results = scraper.load_craigslist_url(url)
scraper.kill()
dictionary_of_listings = scraper.organizeResults(results)
scraper.to_csv(dictionary_of_listings)

Choose file name depend on parameter in command line running pytest

I have a project with structure the same as this:
https://github.com/RomanBaggins/Test_automatisation_final_project.
In conftest.py i get parameters from cmd.
In one of the functions in mane_page.py I need to assign a value(= parameter in cmd) to a variable.
For example:
cmd: pytest -s -v --language=en --status=empty test_main_page.py
import pytest
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def pytest_addoption(parser):
parser.addoption('--language', action='store', default=None, help="Choose language")
parser.addoption('--status', action='store', default=None, help="Choose status")
#pytest.fixture(scope="function")
def browser(request):
user_language = request.config.getoption("language")
if user_language:
options = Options()
options.add_experimental_option('prefs', {'intl.accept_languages': user_language})
browser = webdriver.Chrome(options=options)
browser.implicitly_wait(5)
yield browser
else:
raise pytest.UsageError("--choose your language again")
browser.quit()
#pytest.fixture
def choose_status(request):
return request.config.getoption("status")
Then in main_page.py I want to do this:
from .base_page import BasePage
from selenium.webdriver.common.by import By
from .locators import MainPageLocators
class MainPage(BasePage):
def go_to_login_page(self):
login_link = self.browser.find_element(By.CSS_SELECTOR, "#login_link")
login_link.click()
**chosen status = status** <-----------
def should_be_login_link(self):
assert self.is_element_present(*MainPageLocators.LOGIN_LINK)
Finally I need this result: chosen_satus = "empty"
How can I do it?

You can send it as a parameter to go_to_login_page() from the test
#pytest.fixture
def choose_status(request):
return request.config.getoption("status")
def test_main_page(choose_status):
MainPage().go_to_login_page(choose_status)
class MainPage(BasePage):
def go_to_login_page(self, status):
# status = empty

Here is the final code, that solve my problem:
conftest.py
#pytest.fixture
def choose_status(request):
return request.config.getoption("status")
def test_main_page(choose_status):
MainPage().go_to_login_page(choose_status)
In main_page.py
class MainPage(BasePage):
def go_to_login_page(self,choose_status):
if choose_status == "clear":
#do smth
In test_main_page:
def test_guest_can_go_to_login_page(browser,choose_status):
link = "someurl"
page = MainPage(browser, link)
page.open()
page.go_to_login_page(choose_status)
page.should_be_login_link()

How to define a global method whose variables can be used in other methods in Python

I'm working on an automation project. I am automating the login page. There is a problem with the login page in the website that I'm working on, i.e, it fails the first time I log in, even if the credentials are correct, and then successfully logs in if I log in the second time. So, to overcome this I have included a try except for method. The code:
driver=webdriver.Chrome("C:\\temp\\chromedriver.exe")
url = 'site-address'
driver.get(url)
driver.maximize_window()
wait = WebDriverWait(driver, 120)
class Demo
def func1()
user=driver.find_element_by_xpath("//")
user.send_keys('Name')
pas = driver.find_element_by_xpath("//")
password.send_keys('pas')
btn = driver .find_element_by_xpath("//")
btn.click()
try:
chk=wait.until(EC.presence_of_element_located((By.ID, 'lop')))
assert check.text == "Some Text"
print('PAGE VERIFIED : ', check.text)
except:
print('LOGIN UN-SUCCESSFUL')
print('Re-Trying...')
user=driver.find_element_by_xpath("//")
user.send_keys('Name')
pas = driver.find_element_by_xpath("//")
pas.send_keys('pas')
btn = driver .find_element_by_xpath("//")
btn.click()
So, as you can see I am repeating the process of entering username, password and click twice, once in the beginning and then the same process in the except block. My question is, how to create a method, call it setup() such that I can initialize the driver, URL, username, password in setup(). And then call the method in func1() and then in the except block. I'm new to Python, not familiar how to return the values.

Python classes have a constructor called __init__ which you can use to pass information to when creating an object of this class. Something like this, even if the separation of logic is not the best. But gives you an idea of the constructor.
class Demo:
def __init__(self, driver, url, username, password):
""" Constructor of our demo class, set local variables"""
self.driver = driver
self.username = username
self.password = password
self.driver.get(url)
self.driver.maximize_window()
self.wait = WebDriverWait(self.driver, 120)
def func1(self, attempt=1):
""" Find elements in the opened page and send username and pasword,
then clikc the button (submit?) """
user = self.driver.find_element_by_xpath("//")
user.send_keys(self.username)
pas = driver.find_element_by_xpath("//")
pas.send_keys(self.password)
btn = driver .find_element_by_xpath("//")
btn.click()
try:
# Wait for element
chk = self.wait.until(EC.presence_of_element_located((By.ID, 'lop')))
assert chk.text == "Some Text"
print('PAGE VERIFIED : ', chk.text)
except:
# General exception ..
if attempt >= 3: # Start with 1.. so 1, 2, 3.
print("FAILED THREE TIMES, GIVING UP")
else:
print('LOGIN UN-SUCCESSFUL')
print('Re-Trying...')
self.func1(attempt + 1)
if __name__ == '__main__':
driver = webdriver.Chrome("C:\\temp\\chromedriver.exe")
url = 'site-address'
TEST = Demo(driver=driver, url=url, username='Name', password='pas')
TEST.func1()

Selenium Webdriver with Python: Avoiding multiple logins in my test

At the moment , my test code is calling sign_in() function every time the test for a particular page passes , I want to login once and complete the test for all the pages so that it takes less time to complete the test. Attached is the code from base class and one of the test page.
//this is the base test class
import unittest2
from selenium import webdriver
class BaseTestClass(unittest2.TestCase):
#classmethod
def setUpClass(cls):
"""Setups the connection to selenium and defines the broswer in use"""
#cls.driver = webdriver.Firefox()
cls.driver = webdriver.Remote("http://bs-sel01.lonres.lan:4444/wd/hub", webdriver.DesiredCapabilities.FIREFOX)
cls.test_instance = cls.my_test_class(cls.driver)
cls.url = cls.driver.current_url
# Window for main test. At this point the first window is always the main window
cls.main_window = cls.driver.window_handles[0]
def setUp(self):
self.verificationErrors = []
self.driver.get(self.url)
#classmethod
def tearDownClass(cls):
cls.driver.quit()
def tearDown(self):
self.assertEqual([], self.verificationErrors)
# After a test fixture we close all browsers except the main window
for handle in self.driver.window_handles:
if handle != self.main_window:
self.driver.switch_to_window(handle)
self.driver.close()
self.driver.switch_to_window(self.main_window)
//this is one of the test pages
from selenium import webdriver
from base_test_class import BaseTestClass
from pages import PAGE_TITLES, ParkingSpacesPage
from public_site_pages import PublicSiteHomePage
#
class TestParkingSpacesPage(BaseTestClass):
my_test_class = PublicSiteHomePage
#classmethod
def setUpClass(cls):
super(TestParkingSpacesPage, cls).setUpClass()
cls.test_instance = cls.test_instance.sign_in(ParkingSpacesPage)
cls.url = cls.driver.current_url
#---------------------------------------------------------------------------------------------------------
def test_property_address_link(self):
"""property address link on Parking space page"""
post_code = self.test_instance.property_address_link()
self.driver.implicitly_wait(800)
self.assertIn("%s - Google Maps" % post_code.replace(u'\xa0', u' '), self.driver.title)
#---------------------------------------------------------------------------------------------------------
def test_postcode_link(self):
"""postcode link on Parking space page"""
post_code = self.test_instance.postcode_link()
self.assertIn("%s - Google Maps" % post_code.replace(u'\xa0', u' '), self.driver.title)
#----------------------------------------------------------------------------------------------------------
def test_agent_name_link(self):
"""agent name link on Parking space page"""
element = self.test_instance.agent_name_link()
#confirm if name of property appears in new window title e.g "Lonres.com: Flat 9, 110 frampton Street, NW10"
self.driver.implicitly_wait(800)
assert ("Lonres.com: %s" % element) in self.driver.title
#--------------------------------------------------------------------------------------------------------------------
#the property needs to have at least one photo
def test_view_pic_link(self):
"""view picture link on Parking space page"""
self.test_instance.view_pic_link()
self.driver.implicitly_wait(800)
self.assertIn(PAGE_TITLES["LonresPhotosPage"], self.driver.title)
#------------------------------------------------------------------------------------------
def test_send_to_mobile(self):
"""Send to mobile link on Parking space page"""
self.test_instance.send_to_mobile_link()
self.driver.implicitly_wait(800)
self.assertIn(PAGE_TITLES["LonresSendMobilePhonePage"], self.driver.title)
#-------------------------------------------------------------------------------------
def test_thumbnail_image_link(self):
"""photo link on Parking space page"""
self.test_instance.image_thumbnail_link()
self.driver.implicitly_wait(800)
self.assertIn(PAGE_TITLES["LonresPhotosPage"], self.driver.title)
#----------------------------------------------------------------------------------------------------------------
def test_send_valid_email(self):
"""This test sends a mail with a valid email address"""
#check if we are on the test environment
environment = 'https://test.lonres.lan'
if environment not in self.driver.current_url:
self.skipTest("This test is built to run on only https://test.lonres.lan")
to_address = 'igba#lonres.com'
index = 1
email_page = self.test_instance.send_email(index)
cc_address = email_page.get_cc_address()
email_page.set_to_address(to_address)
email_subject = email_page.get_email_subject()
message_payload = email_page.send_email()
self.assertIn('Lonres.com: Email Sent', self.driver.title)
#inspect message payload
self.assertEquals(to_address, message_payload['to'])
self.assertEquals('Selenium Tester <igbaujege.lonrescomlimited#agentparticulars.com>', message_payload['from'])
self.assertEquals(cc_address, message_payload['cc'])
self.assertEquals(email_subject, message_payload['subject'])

Are you sure you want to do this? Self contained atomic tests are easier to debug and understand failures.
Tests which rely on other tests to leave the system in the right state are brittle and unreliable.
If test runtime is an issue, perhaps you should consider parallel running of your tests using Selenium Grid?

You can create a global boolean hasSignedIn = false
And then in each test, sign in only if necessary.
if not (hasSignedIn):
sign_in()
Then you can avoid signing in whenever possible, without depending on other tests. It takes only milliseconds to check the boolean, so that should save time.