I'm using the python requests module to handle requests on a particular website i'm crawling. I'm fairly new to HTTP requests, but I do understand the basics. Here's the situation. There's a form I want to submit and I do that by using the post method from the requests module:
# I create a session
Session = requests.Session()
# I get the page from witch I'm going to post the url
Session.get(url)
# I create a dictionary containing all the data to post
PostData = {
'param1': 'data1',
'param2': 'data2',
}
# I post
Session.post(SubmitURL, data=PostData)
Is there a any ways to check if the data has been successfully posted?
Is the .status_code method a good way to check that?
If you pick up the result from when you post you can then check the status code:
result = Session.post(SubmitURL, data=PostData)
if result.status_code == requests.codes.ok:
# All went well...
I am a Python newbie but I think the easiest way is:
if response.ok:
# whatever
because all 2XX codes are successful requests not just 200
I am using this code:
import json
def post():
return requests.post('http://httpbin.org/post', data={'x': 1, 'y': 2})
def test_post(self):
txt = post().text
txt = json.loads(txt)
return (txt.get("form") == {'y': '2', 'x': '1'})
For POST responses, the response status code is usually 201 (especially with RestAPIs) but can be 200 sometimes as well depending on how the API has been implemented.
There are a few different solutions:
Solution #1 (use the framework to raise an error) - Recommended:
from requests.exceptions import HTTPError
try:
response = Session.post(SubmitURL, data=PostData)
# No exception will raised if response is successful
response.raise_for_status()
except HTTPError:
# Handle error
else:
# Success
Solution #2 (explicitly check for 200 and 201):
from http import HttpStatus
response = Session.post(SubmitURL, data=PostData)
if response.status_code in [HttpStatus.OK, HttpStatus.CREATED]:
# Success
Solution #3 (all 200 codes are considered successful):
from http import HttpStatus
response = Session.post(SubmitURL, data=PostData)
if response.status_code // 100 == 2:
# Success
Related
As I cannot see any reference on how to use properly the sysparm_query_category, may I confirm on how did you append the sysparm_query_category in get request?
Added below line of codes. I hope someone can help and guide me as I'm new to ServiceNow. Thanks!
from pysnow import Client
...
resource = self.client.resource(api_path=F"/table/{table}")
resource.parameters.add_custom({"sysparm_query_category": "cat_read_generic"})
request = resource.get(query=query, fields=fields)
https://developer.servicenow.com/dev.do#!/reference/api/sandiego/rest/c_TableAPI
I am not aware pysnow API. I hope this helps. Else refer to the comment above.
#Need to install requests package for python
#easy_install requests
import requests
# Set the request parameters
url = 'https://demonightlycoe.service-now.com/api/now/table/incident?sysparm_query=category%3Dcat_read_generic&sysparm_limit=1'
# Eg. User name="admin", Password="admin" for this code sample.
user = 'admin'
pwd = 'admin'
# Set proper headers
headers = {"Content-Type":"application/json","Accept":"application/json"}
# Do the HTTP request
response = requests.get(url, auth=(user, pwd), headers=headers )
# Check for HTTP codes other than 200
if response.status_code != 200:
print('Status:', response.status_code, 'Headers:', response.headers, 'Error Response:',response.json())
exit()
# Decode the JSON response into a dictionary and use the data
data = response.json()
print(data)
Upon checking and testing, we can use the add_custom
client.parameters.add_custom({'foo': 'bar'})
that would be additional to the parameters like &foo=bar
https://pysnow.readthedocs.io/en/latest/usage/parameters.html
Using a simple Python script, i want to send a request, with Python-Requests, to a Django view. The Django view should receive the json data inside the request and should print it to my console; here is what i tried:
This is how i send the request:
url = 'http://127.0.0.1:8000/myview/view'
client = requests.session()
csrftoken = requests.get(url).cookies['csrftoken']
data = json.dumps({'data': 'test-value'})
header = {'X-CSRFToken': csrftoken}
cookies = {'csrftoken': csrftoken}
resp = requests.post(url, data=data, headers=header, cookies=cookies)
And this is how the Django view receives it:
def myview(request):
if request.method == 'POST':
data = request.POST.get('data')
print(data)
print('received.')
response = HttpResponse(get_token(request))
return response
The problem with my current code is that print(data) will throw the following output:
None
received.
[06/Jan/2020 21:23:57] "POST /myview/view HTTP/1.1" 200 64
So, instead of printing test-value, it prints nothing. I don't understand whether the error is in my Django view or in how i'm sending the request. Any advice is appreciated!
The problem is with your request, and entirely caused by this line:
data = json.dumps({'data': 'test-value'})
You simply want
data = {'data': 'test-value'}
The POST data should be sent as a simple dictionary, not a JSON string - see the documentation and example here.
I have the following code for interacting with pull requests on the github api.
def merge(pull):
url = "https://api.github.com/repos/{}/{}/pulls/{}/merge".format(os.environ.get("GITHUB_USERNAME"), os.environ.get("GITHUB_REPO"), pull['number'])
response = requests.put(url, auth=get_auth(), data={})
if response.status_code == 200:
#Merge was successful
return True
else:
#Something went wrong. Oh well.
return response.status_code
def close(pull):
url = "https://api.github.com/repos/{}/{}/pulls/{}".format(os.environ.get("GITHUB_USERNAME"), os.environ.get("GITHUB_REPO"), pull['number'])
payload = {"state" : "closed"}
response = requests.put(url, auth=get_auth(), data=payload)
if response.status_code == 200:
#Close was successful
return True
else:
#Something went wrong. Oh well.
return response.status_code
Now merge works just fine, when I run it with a pull request, the pull request is merged and it feels good.
But close gives me a 404. This is strange since merge can clearly find the pull request, and also shows that I clearly have permissions set up properly so I can close the request.
I have also confirmed that I can close the request manually by logging in on github and pressing the 'close pull request' button.
Why does github give me a 404 for the close function but not the merge function? What is different between these two functions?
The answer is that the 'update a pull request' api call should be a POST request, not a put request.
Changing
response = requests.put(url, auth=get_auth(), data=payload)
to
response = requests.post(url, auth=get_auth(), data=payload)
Fixed the issue.
I'm currently using the python requests library to interact with an external api which uses json.
Each endpoint works via a method (of the api class) and uses the collect_data method.
However I want the scraper to continue running whenever it encounters a http error (and ideally output this to a log).
What's the best way to do this as currently it just breaks when I use http.raise_for_status()
It seems like I should be using a try/except in someway but not sure how best to do this here?
def scrape_full_address(self, house_no, postcode):
address_path = '/api/addresses'
address_url = self.api_source + address_path
payload = {
'houseNo': house_no,
'postcode': postcode,
}
return self.collect_data(url=address_url, method='get', payload=payload)
def collect_data(self, url, method, payload=None):
if method == 'get':
data = None
params = payload
elif method == 'post':
params = None
data = payload
response = getattr(requests, method)(url=url, params=params, json=data, headers=self.headers)
if response.status_code == 200:
return response.json()
else:
return response.raise_for_status()
When you call scrape_full_address() elsewhere in your code wrap that in a try statement.
For more info see: https://wiki.python.org/moin/HandlingExceptions
try:
scrape_full_address(659, 31052)
except HTTPError:
print "Oops! That caused an error. Try again..."
I am trying to parse a response.text that I get when I make a request using the Python Requests library. For example:
def check_user(self):
method = 'POST'
url = 'http://localhost:5000/login'
ck = cookielib.CookieJar()
self.response = requests.request(method,url,data='username=test1&passwd=pass1', cookies=ck)
print self.response.text
When I execute this method, the output is:
{"request":"POST /login","result":"success"}
I would like to check whether "result" equals "success", ignoring whatever comes before.
The manual suggests: if self.response.status_code == requests.codes.ok:
If that doesn't work:
if json.loads(self.response.text)['result'] == 'success':
whatever()
Since the output, response, appears to be a dictionary, you should be able to do
result = self.response.json().get('result')
print(result)
and have it print
'success'
If the response is in json you could do something like (python3):
import json
import requests as reqs
# Make the HTTP request.
response = reqs.get('http://demo.ckan.org/api/3/action/group_list')
# Use the json module to load CKAN's response into a dictionary.
response_dict = json.loads(response.text)
for i in response_dict:
print("key: ", i, "val: ", response_dict[i])
To see everything in the response you can use .__dict__:
print(response.__dict__)
import json
def check_user(self):
method = 'POST'
url = 'http://localhost:5000/login'
ck = cookielib.CookieJar()
response = requests.request(method,url,data='username=test1&passwd=pass1', cookies=ck)
#this line converts the response to a python dict which can then be parsed easily
response_native = json.loads(response.text)
return self.response_native.get('result') == 'success'
I found another solution. It is not necessary to use json module. You can create a dict using dict = eval(whatever) and return, in example, dict["result"]. I think it is more elegant. However, the other solutions also work and are correct
Put in the return of your method like this:
return self.response.json()
If you wanna looking for more details, click this following link:
https://www.w3schools.com/python/ref_requests_response.asp
and search for json() method.
Here is an code example:
import requests
url = 'https://www.w3schools.com/python/demopage.js'
x = requests.get(url)
print(x.json())
In some cases, maybe the response would be as expected. So It'd be great if we can built a mechanism to catch and log the exception.
import requests
import sys
url = "https://stackoverflow.com/questions/26106702/how-do-i-parse-a-json-response-from-python-requests"
response = requests.get(url)
try:
json_data = response.json()
except ValueError as exc:
print(f"Exception: {exc}")
# to find out why you have got this exception, you can see the response content and header
print(str(response.content))
print(str(response.headers))
print(sys.exc_info())
else:
if json_data.get('result') == "success":
# do whatever you want
pass