I'm building a personal webscraper and I'm still in dev phase, but I want to start saving data. The problem is, I cannot PUT or POST from the notebook level, which also means I cannot iterate through a big list of dictionaries/json objects.
I can however to it manually via Postman by just pasting the body and sending it.
Below is my code, which currently returns:
The pastebin URL is:{"message": "Could not parse request body into json: Unrecognized token 'ID': was expecting 'null', 'true', 'false' or NaN\n at [Source: (byte[])"ID=0&District=London"; line: 1, column: 4]"}
import requests
# defining the api-endpoint
API_ENDPOINT = ##############
# data to be sent to api
body = {
"ID": 0,
"District": "London"
}
# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT, data = body)
# extracting response text
pastebin_url = r.text
print("The pastebin URL is:%s"%pastebin_url)
Related question - can I use urllib instead of requests here?
You can try the following:
r = requests.post(url = API_ENDPOINT, json = body)
or
import json
r = requests.post(url = API_ENDPOINT, headers={"Content-Type":"application/json"}, data = json.dumps(body))
Related
I have found this following API endpoint from binance which unfortunately doesn't have a documentation on their page. They are using it directly on their website UI.
I have found the POST Request to the https://www.binance.com/gateway-api/v1/public/future/leverage/token/basket/list actually works. The problem is I would like to convert this info into a dataframe.
I have tried the following, unfortunately, I cannot find a way to convert the following into dataframe as the type is not yet json.
Any help would be appreciated. Thanks in advance
My attempt:
import requests
from requests.structures import CaseInsensitiveDict
url = "https://www.binance.com/gateway-api/v1/public/future/leverage/token/basket/list"
headers = CaseInsensitiveDict()
headers["Content-Type"] = "application/json"
data = """
{"asset": "BTCUP", "current": 1, "size": 20}
"""
resp = requests.post(url, headers=headers, data=data)
print(resp.content) #shows the data as class 'byte'
I'm trying to download a excel file using a python request.
Below is the actual payload pass by web browser when I download required excel file manually.
I'm using below python code to do the same via automation.
import requests
import json
s = requests.Session()
response = s.get(url)
if response.status_code!=200:
return 'loading error!'
collectionId = response.content['collectionId']
payload = {
"collectionId": collectionId
}
response = s.post(url, json=payload)
Here I'm getting status code 400.
Can someone please help me to set payload to match with below snapshot.
I am trying to scrape table values from this website.. There are multiple dropdown which change based on selections in the previous dropdown. Upon inspecting the source, it seems to sending HTTP requests and render the results. However, I can't seem to write script to send those requests myself and scrape the data.
This is what I tried:
import requests
URL = 'https://voterlist.election.gov.np/bbvrs1/index_process_1.php' #API URL
payload = 'vdc=5298&ward=1&list_type=reg_centre' #Unique payload fetched from the network request
response = requests.post(URL,data=payload,verify=False) #POST request to get the data using URL and Payload information
print(response.text)
This is not giving me the expected response which should be a table containing the values. What could be the best approach to take in this case?
So after a few hours of digging in I found an answer. All I had to do was send a cookie while making the request and also change the format the payload was sent in.
headers = {
'num': number, # This the cookie that should be sent which happens to be a session id in my case.
}
payload = {
'state': "1",
"district": "14",
"vdc_mun": "5132",
"ward": "3",
"reg_centre" :""
}
#Then send a post request to the url
res = session.post(url, data=payload, headers=headers, verify=False)
I have been trying to do HTTP post request on the API of this website:
http://text-processing.com/docs/sentiment.html
but it always gives me an error saying that a field is required even if I did mention the key:
import requests
url = 'http://text-processing.com/api/sentiment/'
myobj = {'text ': 'I am feeling reat'}
x = requests.post(url, data = myobj)
print(x.text)
the reponse:
<Response [200]>
Form Validation Errors
text: This field is required.
Does anyone see the problem? because when I do the request using curl it works fine:
curl -d "text=great" http://text-processing.com/api/sentiment/
Response:
{"probability": {"neg": 0.30135019761690551, "neutral": 0.27119050546800266, "pos": 0.69864980238309449}, "label": "pos"}
Remove the space after "text", this will work
import requests
url = 'http://text-processing.com/api/sentiment/'
myobj = {'text': 'I am feeling reat'}
x = requests.post(url, data = myobj)
print(x.text)
rocksteady's solution worked
He did originally refer to dictionaries. But the following code to send the JSON string also worked wonders using requests:
import requests
headers = {
'Authorization': app_token
}
url = api_url + "/b2api/v1/b2_get_upload_url"
content = json.dumps({'bucketId': bucket_id})
r = requests.post(url, data = content, headers = headers)
I'm working with an API that requires me to send JSON as a POST request to get results. Problem is that Python 3 won't allow me to do this.
The following Python 2 code works fine, in fact it's the official sample:
request = urllib2.Request(
api_url +'/b2api/v1/b2_get_upload_url',
json.dumps({ 'bucketId' : bucket_id }),
headers = { 'Authorization': account_authorization_token }
)
response = urllib2.urlopen(request)
However, using this code in Python 3 only makes it complain about data being invalid:
import json
from urllib.request import Request, urlopen
from urllib.parse import urlencode
# -! Irrelevant code has been cut out !-
headers = {
'Authorization': app_token
}
url = api_url + "/b2api/v1/b2_get_upload_url"
# Tested both with encode and without
content = json.dumps({'bucketId': bucket_id}).encode('utf-8')
request = Request(
url=url,
data=content,
headers=headers
)
response = urlopen(req)
I've tried doing urlencode(), like you're supposed to. But this returns a 400 status code from the web server, because it's expecting pure JSON. Even if the pure JSON data is invalid, I need to somehow force Python into sending it.
EDIT: As requested, here are the errors I get. Since this is a flask application, here's a screenshot of the debugger:
Screenshot
Adding .encode('utf-8') gives me an "Expected string or buffer" error
EDIT 2: Screenshot of the debugger with .encode('utf-8') added
Since I have a similar application running, but the client still was missing, I tried it myself.
The server which is running is from the following exercise:
Miguel Grinberg - designing a restful API using Flask
That's why it uses authentication.
But the interesting part: Using requests you can leave the dictionary as it is.
Look at this:
username = 'miguel'
password = 'python'
import requests
content = {"title":"Read a book"}
request = requests.get("http://127.0.0.1:5000/api/v1.0/projects", auth=(username, password), params=content)
print request.text
It seems to work :)
Update 1:
POST requests are done using requests.post(...)
This here describes it well : python requests
Update 2:
In order to complete the answer:
requests.post("http://127.0.0.1:5000/api/v1.0/projects", json=content)
sends the json-string.
json is a valid parameter of the request and internally uses json.dumps()...