I'm currently testing my Django application in order to add some CI/CD to it. However, most of my views contain an AJAX section for requests sent by the frontend. I saw that for testing those I can only just do something like this:
response: HttpResponseBase = self.client.post(
path=self.my_page_url,
content_type='application/json',
HTTP_X_REQUESTED_WITH='XMLHttpRequest',
data={
'id': '123456',
'operation': "Fill Details"
}
)
The XMLHttpRequest is making most of the magic here (I think), by simulating the headers that an AJAX request would have. However, in my view I have a section where I do: request.POST['operation'], but this seems to fail during tests since apparently no data is passed through the POST attribute. Here's the code of the view that I'm using right now:
MyView(request):
is_ajax: bool = request.headers.get('x-requested-with') == 'XMLHttpRequest'
if is_ajax:
operation = request.POST['operation']
I checked and my data is being passed in request.body. I could include an or statement, but it would be ideal if the code for views was not modified because of tests. Is there any way to get the client.post method to pass the data through the POST attribute?
You can simulate ajax like POST using the python requests library.
import requests
headers = {
'X-Requested-With': 'XMLHttpRequest',
'Content-Type': 'application/x-www-form-urlencoded',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36',
}
data = {
'id': '123456',
'operation': "Fill Details"
}
session = requests.Session()
session.post(url, data=data, headers=headers)
Related
I am currently using Python requests to scrape data from a website and using Postman as a tool to help me do it.
To those not familiar with Postman, it sends a get request and generates a code snippet to be used in many languages, including Python.
By using it, I can get data from the website quite easily, but it seems as like the 'Cookie' aspect of headers provided by Postman changes with time, so I can't automate my code to run anytime. The issue is that when the cookie is not valid I get an access denied message.
Here's an example of the code provided by Postman:
import requests
url = "https://wsloja.ifood.com.br/ifood-ws-v3/restaurants/7c854a4c-01a4-48d8-b3d4-239c6c069f6a/menu"
payload = {}
headers = {
'access_key': '69f181d5-0046-4221-b7b2-deef62bd60d5',
'browser': 'Windows',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36',
'Accept': 'application/json, text/plain, */*',
'secret_key': '9ef4fb4f-7a1d-4e0d-a9b1-9b82873297d8',
'Cache-Control': 'no-cache, no-store',
'X-Ifood-Session-Id': '85956739-2fac-4ebf-85d3-1aceda9738df',
'platform': 'Desktop',
'app_version': '8.37.0',
'Cookie': 'session_token=TlNUXzMyMjJfMTU5Nzg1MDE5NTIxNF84NDI5NTA2NDQ2MjUxMg==; _abck=AD1745CB8A0963BF3DD67C8AF7932007~-1~YAAQtXsGYH8UUe9zAQAACZ+IAgStbP4nYLMtonPvQ+4UY+iHA3k6XctPbGQmPF18spdWlGiDB4/HbBvDiF0jbgZmr2ETL8YF+f71Uwhsj+L8K+Fk4PFWBolAffkIRDfSubrf/tZOYRfmw09o59aFuQor5LeqxzXkfVsXE8uIJE0P/nC1JfImZ35G0OFt+HyIgDUZMFQ54Wnbap7+LMSWcvMKF6U/RlLm46ybnNnT/l/NLRaEAOIeIE3/JdKVVcYT2t4uePfrTkr5eD499nyhFJCwSVQytS9P7ZNAM4rFIPnM6kPtwcPjolLNeeU=~-1~-1~-1; ak_bmsc=129F92B2F8AC14A400433647B8C29EA3C9063145805E0000DB253D5F49CE7151~plVgguVnRQTAstyzs8P89cFlKQnC9ISQCH9KPHa8xYPDVoV2iQ/Hij2PL9r8EKEqcQfzkGmUWpK09ZpU0tL/llmBloi+S+Znl5P5/NJeV6Ex2gXqBu1ZCxc9soMWWyrdvG+0FFvSP3a6h3gaouPh2O/Tm4Ghk9ddR92t380WBkxvjXBpiPzoYp1DCO4yrEsn3Tip1Gan43IUHuCvO+zkRmgrE3Prfl1T/g0Px9mvLSVrg=; bm_sz=3106E71C2F26305AE435A7DA00506F01~YAAQRTEGyfky691zAQAAGuDbBggFW4fJcnF1UtgEsoXMFkEZk1rG8JMddyrxP3WleKrWBY7jA/Q08btQE43cKWmQ2qtGdB+ryPtI2KLNqQtKM5LnWRzU+RqBQqVbZKh/Rvp2pfTvf5lBO0FRCvESmYjeGvIbnntzaKvLQiDLO3kZnqmMqdyxcG1f51aoOasrjfo=; bm_sv=B4011FABDD7E457DDA32CBAB588CE882~aVOIuceCgWY25bT2YyltUzGUS3z5Ns7gJ3j30i/KuVUgG1coWzGavUdKU7RfSJewTvE47IPiLztXFBd+mj7c9U/IJp+hIa3c4z7fp22WX22YDI7ny3JxN73IUoagS1yQsyKMuxzxZOU9NpcIl/Eq8QkcycBvh2KZhhIZE5LnpFM='
}
response = requests.request("GET", url, headers=headers, data = payload)
print(response.text.encode('utf8'))
Here's just the Cookie part where I get access denied:
'Cookie': 'session_token=TlNUXzMyMjJfMTU5Nzg1MDE5NTIxNF84NDI5NTA2NDQ2MjUxMg==; _abck=AD1745CB8A0963BF3DD67C8AF7932007~-1~YAAQtXsGYH8UUe9zAQAACZ+IAgStbP4nYLMtonPvQ+4UY+iHA3k6XctPbGQmPF18spdWlGiDB4/HbBvDiF0jbgZmr2ETL8YF+f71Uwhsj+L8K+Fk4PFWBolAffkIRDfSubrf/tZOYRfmw09o59aFuQor5LeqxzXkfVsXE8uIJE0P/nC1JfImZ35G0OFt+HyIgDUZMFQ54Wnbap7+LMSWcvMKF6U/RlLm46ybnNnT/l/NLRaEAOIeIE3/JdKVVcYT2t4uePfrTkr5eD499nyhFJCwSVQytS9P7ZNAM4rFIPnM6kPtwcPjolLNeeU=~-1~-1~-1; ak_bmsc=129F92B2F8AC14A400433647B8C29EA3C9063145805E0000DB253D5F49CE7151~plVgguVnRQTAstyzs8P89cFlKQnC9ISQCH9KPHa8xYPDVoV2iQ/Hij2PL9r8EKEqcQfzkGmUWpK09ZpU0tL/llmBloi+S+Znl5P5/NJeV6Ex2gXqBu1ZCxc9soMWWyrdvG+0FFvSP3a6h3gaouPh2O/Tm4Ghk9ddR92t380WBkxvjXBpiPzoYp1DCO4yrEsn3Tip1Gan43IUHuCvO+zkRmgrE3Prfl1T/g0Px9mvLSVrg=; bm_sz=3106E71C2F26305AE435A7DA00506F01~YAAQRTEGyfky691zAQAAGuDbBggFW4fJcnF1UtgEsoXMFkEZk1rG8JMddyrxP3WleKrWBY7jA/Q08btQE43cKWmQ2qtGdB+ryPtI2KLNqQtKM5LnWRzU+RqBQqVbZKh/Rvp2pfTvf5lBO0FRCvESmYjeGvIbnntzaKvLQiDLO3kZnqmMqdyxcG1f51aoOasrjfo=; bm_sv=B4011FABDD7E457DDA32CBAB588CE882~aVOIuceCgWY25bT2YyltUzGUS3z5Ns7gJ3j30i/KuVUgG1coWzGavUdKU7RfSJewTvE47IPiLztXFBd+mj7c9U/IJp+hIa3c4z7fp22WX23E755znZL76c0V/amxbHU9BUnrEff3HGcsniyh5mU+C9XVmtNRLd8oT1UW9WUg3qE=' }
Which is slightly different from the one before.
How could I get through this by somehow having python get the session token?
Apparently just removing 'Cookie' from headers does the job.
I am trying to login to a site called grailed.com and follow a certain product. The code below is what I have tried.
The code below succeeds in logging in with my credentials. However whenever I try to follow a product (the id in the payload is the id of the product) the code runs without any errors but fails to follow the product. I am confused at this behavior. Is it a similar case to Instagram (where Instagram blocks any attempt to interact programmatically with their site and force you to use their API (grailed.com does not have a API for the public to use AFAIK)
I tried the following code (which looks exactly like the POST request sent when you follow on the site).
headers/data defined here
r = requests.Session()
v = r.post("https://www.grailed.com/api/sign_in", json=data,headers = headers)
headers = {
'authority': 'www.grailed.com',
'method': 'POST',
"path": "/api/follows",
'scheme': 'https',
'accept': 'application/json',
'accept-encoding': 'gzip, deflate, br',
"content-type": "application/json",
"x-amplitude-id": "1547853919085",
"x-api-version": "application/grailed.api.v1",
"x-csrf-token": "9ph4VotTqyOBQzcUt8c3C5tJrFV7VlT9U5XrXdbt9/8G8I14mGllOMNGqGNYlkES/Z8OLfffIEJeRv9qydISIw==",
"origin": "https://www.grailed.com",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"
}
payload = {
"id": "7917017"
}
b = r.post("https://www.grailed.com/api/follows",json = payload,headers = headers)
If API is not designed to be public, you are most likely missing csrf token in your follow headers.
You have to find an CSRF token, and add it to /api/follows POST.
taking fast look at code, this might be hard as everything goes inside javascript.
I am emulating the requests sent to this website to try add a product to cart - although it is not working as intended and I am not sure why. Here is my series of requests sent.
s = requests.Session()
payload = {
"sku": "182418M20400102",
"serviceType": "product-details",
"userId": None,
}
headers = {
'content-type': 'application/json',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
}
s.get("https://www.ssense.com/en-ca/mini-shopping-bag") // Initialize cookies
resp = s.post("https://www.ssense.com/en-ca/api/shopping-bag/182418M20400102", json=payload, headers=headers)
print(resp.status_code)
bag = s.get("https://www.ssense.com/en-us/mini-shopping-bag")
print(bag.json())
Console printout is:
https://www.ssense.com/en-ca/api/shopping-bag/182418M20400102 // pid in this case
204 // Expected status code
{'quantity': 0, 'token': 'xxxx'} // qty should be 1
Not sure why it is not working.
you will have to do cookie management. this website stores your value in a cookie.
since you are making a post request, the next time you hit a get request, the values are stored in a cookie in the actual website. The web browsers manage our cookies so the bag has the data added. If you hit the same in incognito mode you will not have any data.
refer http://docs.python-requests.org/en/master/user/advanced/ for setting a cookie.
I'm trying to login to a webpage using python 3 using requests and lxml. However, after sending a post request to the login page, I can't enter pages that are available after login. What am I missing?
import requests
from lxml import html
session_requests = requests.session()
login_URL = 'https://www.voetbal.nl/inloggen'
r = session_requests.get(login_URL)
tree = html.fromstring(r.text)
form_build_id = list(set(tree.xpath("//input[#name='form_build_id']/#value")))[0]
payload = {
'email':'mom.soccer#mail.com',
'password':'testaccount',
'form_build_id':form_build_id
}
headers = {
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate, br',
'Accept-Language':'nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7',
'Cache-Control':'max-age=0',
'Connection':'keep-alive',
'Content-Type':'multipart/form-data; boundary=----WebKitFormBoundarymGk1EraI6yqTHktz',
'Host':'www.voetbal.nl',
'Origin':'https://www.voetbal.nl',
'Referer':'https://www.voetbal.nl/inloggen',
'Upgrade-Insecure-Requests':'1',
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
}
result = session_requests.post(
login_URL,
data = payload,
headers = headers
)
pvc_url = 'https://www.voetbal.nl/club/BBCB10Z/overzicht'
result_pvc = session_requests.get(
pvc_url,
headers = headers
)
print(result_pvc.text)
The account in this sample is activated, but it is just a test-account which I created to put my question up here. Feel free to try it out.
Answer:
there where multiple problems:
Payload: 'form_id': 'voetbal_login_login_form' was missing. Thanks #t.m.adam
Cookies: request cookies where missing. They seem to be static, so I tried to add them manually, which worked. Thanks #match and #Patrick Doyle
Headers: removed the 'content-type' line; which contained a dynamic part.
Login works like a charm now!
I am trying to remotely trigger actions on a website via post requests.
Here is an example of the request that is performed when I click the button on the website : http://pastebin.com/raw.php?i=dd3sfzpc
And here is what I send : http://pastebin.com/raw.php?i=eqsBDKUp.
I use the following code :
import time
import requests
import json
import urllib
session = requests.session()
login_data = {
'Username': #the mail here,
'Password': #the pass here,
'submit': 'login',
}
res = session.post(#[the url where I log in], data=login_data)
rep=session.get(#[the url from which I get the session s data]).text
#below I fetch some data I need in the post request
p=rep.find('id="__EVENTVALIDATION"')
eventval=rep[rep.find('value="',p)+len('value="'):rep.find('" />',p)]
p=rep.find('id="__VIEWSTATE"')
viewstate=rep[rep.find('value="',p)+len('value="'):rep.find('" />',p)]
p=rep.find('id="__VIEWSTATEGENERATOR"')
viewstategenerator=rep[rep.find('value="',p)+len('value="'):rep.find('" />',p)]
# parameters of the action
current_trade_currency="tx"
have="Robux" if current_trade_currency=="tx" else "Tickets"
want="Tickets" if current_trade_currency=="tx" else "Robux"
fromCur=10
toCur=100
tab={}
tab["__LASTFOCUS"]=""
tab["__EVENTTARGET"]="ctl00$ctl00$cphRoblox$cphMyRobloxContent$ctl00$SubmitTradeButton"
tab["__EVENTARGUMENT"]=""
tab["comments"]=""
tab["__VIEWSTATE"]=viewstate
tab["__EVENTVALIDATION"]=eventval
tab["__VIEWSTATEGENERATOR"]=viewstategenerator
tab["rdoNotifications"]="on"
tab["ctl00$ctl00$cphRoblox$cphMyRobloxContent$ctl00$OrderType"]="LimitOrderRadioButton"
tab["ctl00$ctl00$cphRoblox$cphMyRobloxContent$ctl00$HaveAmountTextBoxRestyle"]=str(fromCur)
tab["ctl00$ctl00$cphRoblox$cphMyRobloxContent$ctl00$HaveCurrencyDropDownList"]=str(have)
tab["ctl00$ctl00$cphRoblox$cphMyRobloxContent$ctl00$WantAmountTextBox"]=str(toCur)
tab["ctl00$ctl00$cphRoblox$cphMyRobloxContent$ctl00$WantCurrencyDropDownList"]=str(want)
tab["ctl00$ctl00$cphRoblox$cphMyRobloxContent$ctl00$AllowSplitTradesCheckBox"]="on"
headers={
'Cache-Control': 'max-age=0',
'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Origin' : 'http://www.roblox.com',
'User-Agent' : 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.53 Safari/537.36',
'Content-Type' : 'application/x-www-form-urlencoded',
'Referer' : 'http://www.roblox.com/My/Money.aspx',
'Accept-Encoding' : 'gzip, deflate',
'Accept-Language' : 'en-US,en;q=0.8,fr;q=0.6'
}
session.post(#[the url where I perform the post request],data=tab).text.encode("utf-8"),headers=headers)
I cannot see any difference between the post request sent when the button is clicked, and the post request I send. However, my post request does not trigger anything.
What is the cause of this ?
Edit : here is a screenshot of some caracteristics of the request made by the website :