Scraping this site - python

Im trying to mimic this POST request from this site with this payload:
from this URL: https://surviv.io/stats/gert1
Here is an image of the request im trying to mimic.
Here is my current code in python:
import requests
headers = {'content-type': 'application/json; charset=UTF-8'}
url = 'https://surviv.io/api/user_stats'
payload = {"slug":"gert1","interval":"all","mapIdFilter":"-1"}
r = requests.post(url=url, headers=headers, data=payload)
print(r.content)
This returns:
b'<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body bgcolor="white">\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'
This is not what I want it return. I want it to return the exact response shown in the response tab of the user_stats requests, which contains the player's stats.
This is what I want it to return:
{"slug":"gert1","username":"GERT","player_icon":"","banned":false,"wins":61,"kills":2830,"games":2034,"kpg":"1.4","modes":[{"teamMode":1,"games":1512,"wins":46,"kills":2230,"winPct":"3.0","mostKills":21,"mostDamage":1872,"kpg":"1.5","avgDamage":169,"avgTimeAlive":92},{"teamMode":2,"games":255,"wins":4,"kills":234,"winPct":"1.6","mostKills":8,"mostDamage":861,"kpg":"0.9","avgDamage":162,"avgTimeAlive":102},{"teamMode":4,"games":267,"wins":11,"kills":366,"winPct":"4.1","mostKills":17,"mostDamage":2225,"kpg":"1.4","avgDamage":246,"avgTimeAlive":125}]}


You should use the json attribute rather than data in the post method. r = requests.post(url=url, headers=headers, json=payload)


Change your code to following your forgot to use json :
import json
import requests
headers = {'content-type': 'application/json; charset=UTF-8'}
url = 'https://surviv.io/api/user_stats'
payload = {"slug":"gert1","interval":"all","mapIdFilter":"-1"}
r = requests.post(url=url, headers=headers, data=json.dumps(payload))
print(r.content)

Related

How does Postman to generate this cookie?

import requests
url = "https://apiexample.com/load/v1/action/aaaaaaaaaaaaa"
payload={}
headers = {
'Authorization': 'OAuth oauth_consumer_key="aaaaaa",oauth_signature_method="HMAC-SHA1",oauth_timestamp="1664837015",oauth_nonce="mKyTFn7OtsV",oauth_version="1.0",oauth_signature="aaaaaaaaaaaa"',
'Cookie': 'JSESSIONID=M7n-S-aGe8asRnTjNOUGowak5i5avsRBx6A4H8au.madsedepre'
}
response = requests.request("GET", url, headers=headers, data=payload)
print(response.text)
I am trying to retrieve information from an api, and I am using postman...
How can I know what postman is doing to generate that cookie??
and how can I generate it using python requests?

401 with a POST using requests

Using Postman, I can make a POST to an endpoint with a JSON body, and it returns be some results.
No authorisation on the endpoint.
Trying to get this to work with requests fails with a 401.
What am I missing here:
Code:
import json
import requests
url = 'https://wabi-australia-southeast-api.analysis.windows.net/public/reports/querydata?synchronous=true'
header = {"content-type": "application/json"}
body = {"version":"1.0.0","queries":[{"Query":{"Commands":[{"SemanticQueryDataShapeCommand":{"Query":{"Version":2,"From":[{"Name":"d1","Entity":"dimLGA","Type":0},{"Name":"l","Entity":"Linelist","Type":0}],"Select":[{"Column":{"Expression":{"SourceRef":{"Source":"d1"}},"Property":"LGAName"},"Name":"dimLGA.LGAName"},{"Measure":{"Expression":{"SourceRef":{"Source":"l"}},"Property":"Cases"},"Name":"Linelist.Cases"}],"Where":[{"Condition":{"Not":{"Expression":{"Comparison":{"ComparisonKind":0,"Left":{"Column":{"Expression":{"SourceRef":{"Source":"d1"}},"Property":"LGAName"}},"Right":{"Literal":{"Value":"null"}}}}}}},{"Condition":{"In":{"Expressions":[{"Column":{"Expression":{"SourceRef":{"Source":"l"}},"Property":"clin_status_n"}}],"Values":[[{"Literal":{"Value":"'Admitted to ICU'"}}],[{"Literal":{"Value":"'Admitted, not known to be in ICU'"}}],[{"Literal":{"Value":"'Home isolation'"}}],[{"Literal":{"Value":"'Hotel detention'"}}],[{"Literal":{"Value":"'Hospital in the home'"}}],[{"Literal":{"Value":"'Under investigation'"}}]]}}}]},"Binding":{"Primary":{"Groupings":[{"Projections":[0,1]}]},"DataReduction":{"DataVolume":3,"Primary":{"Window":{"Count":500}}},"Version":1}}}]},"CacheKey":"{\"Commands\":[{\"SemanticQueryDataShapeCommand\":{\"Query\":{\"Version\":2,\"From\":[{\"Name\":\"d1\",\"Entity\":\"dimLGA\",\"Type\":0},{\"Name\":\"l\",\"Entity\":\"Linelist\",\"Type\":0}],\"Select\":[{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"d1\"}},\"Property\":\"LGAName\"},\"Name\":\"dimLGA.LGAName\"},{\"Measure\":{\"Expression\":{\"SourceRef\":{\"Source\":\"l\"}},\"Property\":\"Cases\"},\"Name\":\"Linelist.Cases\"}],\"Where\":[{\"Condition\":{\"Not\":{\"Expression\":{\"Comparison\":{\"ComparisonKind\":0,\"Left\":{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"d1\"}},\"Property\":\"LGAName\"}},\"Right\":{\"Literal\":{\"Value\":\"null\"}}}}}}},{\"Condition\":{\"In\":{\"Expressions\":[{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"l\"}},\"Property\":\"clin_status_n\"}}],\"Values\":[[{\"Literal\":{\"Value\":\"'Admitted to ICU'\"}}],[{\"Literal\":{\"Value\":\"'Admitted, not known to be in ICU'\"}}],[{\"Literal\":{\"Value\":\"'Home isolation'\"}}],[{\"Literal\":{\"Value\":\"'Hotel detention'\"}}],[{\"Literal\":{\"Value\":\"'Hospital in the home'\"}}],[{\"Literal\":{\"Value\":\"'Under investigation'\"}}]]}}}]},\"Binding\":{\"Primary\":{\"Groupings\":[{\"Projections\":[0,1]}]},\"DataReduction\":{\"DataVolume\":3,\"Primary\":{\"Window\":{\"Count\":500}}},\"Version\":1}}}]}","QueryId":"","ApplicationContext":{"DatasetId":"5b547437-24c9-4b22-92de-900b3b3f4785","Sources":[{"ReportId":"964ef513-8ff4-407c-8068-ade1e7f64ca5"}]}}],"cancelQueries":[],"modelId":1959902}
r = requests.post(url, data=json.dumps(body), headers=header)
print(r.status_code)
print(r.raise_for_status())
This returns a 401:
HTTPError: 401 Client Error: Unauthorized for url:
https://wabi-australia-southeast-api.analysis.windows.net/public/reports/querydata?synchronous=true

Just found out Postman lets you generate a python code snippet for Requests. Amazing.
This solved my problem.
import requests
url = "https://wabi-australia-southeast-api.analysis.windows.net/public/reports/querydata"
payload = "{\"version\":\"1.0.0\",\"queries\":[{\"Query\":{\"Commands\":[{\"SemanticQueryDataShapeCommand\":{\"Query\":{\"Version\":2,\"From\":[{\"Name\":\"d1\",\"Entity\":\"dimLGA\",\"Type\":0},{\"Name\":\"l\",\"Entity\":\"Linelist\",\"Type\":0}],\"Select\":[{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"d1\"}},\"Property\":\"LGAName\"},\"Name\":\"dimLGA.LGAName\"},{\"Measure\":{\"Expression\":{\"SourceRef\":{\"Source\":\"l\"}},\"Property\":\"Cases\"},\"Name\":\"Linelist.Cases\"}],\"Where\":[{\"Condition\":{\"Not\":{\"Expression\":{\"Comparison\":{\"ComparisonKind\":0,\"Left\":{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"d1\"}},\"Property\":\"LGAName\"}},\"Right\":{\"Literal\":{\"Value\":\"null\"}}}}}}},{\"Condition\":{\"In\":{\"Expressions\":[{\"Column\":{\"Expression\":{\"SourceRef\":{\"Source\":\"l\"}},\"Property\":\"clin_status_n\"}}],\"Values\":[[{\"Literal\":{\"Value\":\"'Admitted to ICU'\"}}],[{\"Literal\":{\"Value\":\"'Admitted, not known to be in ICU'\"}}],[{\"Literal\":{\"Value\":\"'Home isolation'\"}}],[{\"Literal\":{\"Value\":\"'Hotel detention'\"}}],[{\"Literal\":{\"Value\":\"'Hospital in the home'\"}}],[{\"Literal\":{\"Value\":\"'Under investigation'\"}}]]}}}]},\"Binding\":{\"Primary\":{\"Groupings\":[{\"Projections\":[0,1]}]},\"DataReduction\":{\"DataVolume\":3,\"Primary\":{\"Window\":{\"Count\":500}}},\"Version\":1}}}]},\"CacheKey\":\"{\\\"Commands\\\":[{\\\"SemanticQueryDataShapeCommand\\\":{\\\"Query\\\":{\\\"Version\\\":2,\\\"From\\\":[{\\\"Name\\\":\\\"d1\\\",\\\"Entity\\\":\\\"dimLGA\\\",\\\"Type\\\":0},{\\\"Name\\\":\\\"l\\\",\\\"Entity\\\":\\\"Linelist\\\",\\\"Type\\\":0}],\\\"Select\\\":[{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"d1\\\"}},\\\"Property\\\":\\\"LGAName\\\"},\\\"Name\\\":\\\"dimLGA.LGAName\\\"},{\\\"Measure\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"l\\\"}},\\\"Property\\\":\\\"Cases\\\"},\\\"Name\\\":\\\"Linelist.Cases\\\"}],\\\"Where\\\":[{\\\"Condition\\\":{\\\"Not\\\":{\\\"Expression\\\":{\\\"Comparison\\\":{\\\"ComparisonKind\\\":0,\\\"Left\\\":{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"d1\\\"}},\\\"Property\\\":\\\"LGAName\\\"}},\\\"Right\\\":{\\\"Literal\\\":{\\\"Value\\\":\\\"null\\\"}}}}}}},{\\\"Condition\\\":{\\\"In\\\":{\\\"Expressions\\\":[{\\\"Column\\\":{\\\"Expression\\\":{\\\"SourceRef\\\":{\\\"Source\\\":\\\"l\\\"}},\\\"Property\\\":\\\"clin_status_n\\\"}}],\\\"Values\\\":[[{\\\"Literal\\\":{\\\"Value\\\":\\\"'Admitted to ICU'\\\"}}],[{\\\"Literal\\\":{\\\"Value\\\":\\\"'Admitted, not known to be in ICU'\\\"}}],[{\\\"Literal\\\":{\\\"Value\\\":\\\"'Home isolation'\\\"}}],[{\\\"Literal\\\":{\\\"Value\\\":\\\"'Hotel detention'\\\"}}],[{\\\"Literal\\\":{\\\"Value\\\":\\\"'Hospital in the home'\\\"}}],[{\\\"Literal\\\":{\\\"Value\\\":\\\"'Under investigation'\\\"}}]]}}}]},\\\"Binding\\\":{\\\"Primary\\\":{\\\"Groupings\\\":[{\\\"Projections\\\":[0,1]}]},\\\"DataReduction\\\":{\\\"DataVolume\\\":3,\\\"Primary\\\":{\\\"Window\\\":{\\\"Count\\\":500}}},\\\"Version\\\":1}}}]}\",\"QueryId\":\"\",\"ApplicationContext\":{\"DatasetId\":\"5b547437-24c9-4b22-92de-900b3b3f4785\",\"Sources\":[{\"ReportId\":\"964ef513-8ff4-407c-8068-ade1e7f64ca5\"}]}}],\"cancelQueries\":[],\"modelId\":1959902}"
headers = {
'Content-Type': 'text/plain'
}
response = requests.request("POST", url, headers=headers, data = payload)
print(response.text.encode('utf8'))

I have some troubles with a post request

I'm trying to scrape a website but i get a 500 response.
It's a relocation website and i'm trying to get prices for different input parameters (volume, destination, departureCity ...).
import requests
from bs4 import BeautifulSoup
url = "https://www.demenagerfacile.com/devis/create"
payload = "volume_set_with_vcalc=false&id_vc_quote=&mode=MOVE_CLASSIC&fromCity=150+Rue+Saint-Maur%2C+Paris%2C+France&locationDeparture=%7B%22lat%22%3A%2248.8690238%22%2C%22lng%22%3A%222.3745715999999675%22%7D&placeIdDeparture=%22ChIJOTRebuRt5kcRoklIrmMZx6Y%22&countryDep=FR&toCity=Rue+de+Marseille%2C+Lyon%2C+France&locationDestination=%7B%22lat%22%3A%2245.7509119%22%2C%22lng%22%3A%224.83963289999997%22%7D&placeIdDestination=%22Eh5SdWUgZGUgTWFyc2VpbGxlLCBMeW9uLCBGcmFuY2UiLiosChQKEgm7NI_oRer0RxEcVzzVtXmrQxIUChIJl4foalHq9EcR8CG75CqrCAQ%22&countryDest=FR&fixedDate=on&date=2019-07-18&volume=30"
headers = {
"content-type": "application/x-www-form-urlencoded",
"cookie": "PHPSESSID=k3f99mg9urimu9dpc1ot7vudf5; __stripe_mid=e5c971f7-5e78-4e61-92db-a62dda48fd2c; _ga=GA1.2.110393239.1561101566; _gid=GA1.2.1597485519.1561101566; previousServerId=0; _fbp=fb.1.1561101566214.588023040; _hjIncludedInSample=1; hubspotutk=e7e9f524c880e1091d415cfc5cbd4903; __hssrc=1; apzContact=%5B%7B%22buttonIdKey%22%3A%229deacb07173c79d76baf4f58c56037ad%22%2C%22apizeeSessionId%22%3A%22334d6232-930f-41ec-aab9-0753c7b3036a%22%2C%22apiconfSession%22%3A%229deacb07173c79d76baf4f58c56037ad%22%2C%22apiconfOfflineMessageState%22%3Anull%7D%5D; cookiebanner-accepted=1; __stripe_sid=5e2fb6df-ea5c-470a-98f3-44e172955633; __hstc=78993675.e7e9f524c880e1091d415cfc5cbd4903.1561101566944.1561101566944.1561106755827.2; _gat_UA-65262710-1=1; __hssc=78993675.3.1561106755827; apiCCId=334d6232-930f-41ec-aab9-0753c7b3036a-0000; apiKey=d7897a4712a9413c3a363e737954d853; sessionId=230a5a70-9401-11e9-935a-19ca5a0a60c1",
}
response = requests.request("POST", url, data=payload)
print(response)

Requests’ simple API means that all forms of HTTP request are as obvious. For example, this is how you make an HTTP POST request
Replace your code
response = requests.request("POST", url, data=payload)
TO
response = requests.post(url,headers=headers, data=payload)

Hi #mhabak1 I just checked and it seems to be working,
Please add headers while posting the request like below,
response = requests.post(url, data=payload, headers=headers)
Thanks

Error posting comments using Facebook Graph API

My aim is to post a comment to a particular post id using Facebook graph API.
This is the code snippet for the same:
url = 'https://graph.facebook.com/v2.11/<post_id>/comments'
parameters = {'access_token': <FACEBOOK_ACCESS_TOKEN>, 'message': 'test comment'}
headers = {"content-type": "application/json"}
parameters = json.dumps(parameters)
response = requests.post(url, data=parameters, headers=headers, timeout=10)
I am calling this API inside my DJANGO POST API.
For Some Reason, Calling the Facebook API through this code doesnt work. The API call gets timeout after 10 seconds.
If I call the Facebook API through Postman / YARC , the comment gets posted successfully.
Can any one tell me where I am going wrong?

Python Requests example:
import requests
url = "https://graph.facebook.com/v2.11/yourPostId/comments"
querystring = {"access_token":"yourtoken"}
payload = "message=test%20comment"
headers = {
'content-type': "application/x-www-form-urlencoded",
'cache-control': "no-cache"
}
response = requests.request("POST", url, data=payload, headers=headers, params=querystring)
print(response.text)
Python http.client example:
import http.client
conn = http.client.HTTPSConnection("graph.facebook.com")
payload = "message=test%20comment"
headers = {
'content-type': "application/x-www-form-urlencoded",
'cache-control': "no-cache"
}
conn.request("POST", "/v2.11/yourPostId/comments?access_token=yourtoken", payload, headers)
res = conn.getresponse()
data = res.read()
print(data.decode("utf-8"))

Python Requests POST url encoded request not working

I am trying to make a post request to a url with Requests in Python. I am getting 500 ERROR message. I think the problem is in not properly encoding in content-type to x-www-form-urlencoded. If I try the same request in Postman it is working.
My Python code:
import requests
def get_pdf_link_by_post(url, payload):
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
r = requests.post(url, data=payload, headers=headers)
print requests
print "**************Headers*******************"
print r.headers
print r.content
def process_kat_causelist(date, adv_name):
url = 'http://kat.kar.nic.in:8080/causelistsearch.jsp'
for court_hall_idx in range(1, 4):
court_hall = str(court_hall_idx)+'~Court Hall '+str(court_hall_idx)
for list_idx in range(1, 7):
list_num = str(list_idx)+'~List '+str(list_idx)
payload = {'firstClDate': date, 'secondCldate': date, 'courthall': court_hall, 'lists': list_num}
print payload
get_pdf_link_by_post(url, payload)
process_kat_causelist('25/08/2015', 'ani')
Postman request:

In the payload secondCldate must be secondClDate

Categories