I am learning python API testing using urllib2 module.I tried to execute the code.but throwing the following msg.Can anybody help me.Thanks in advance.
code:
url = "http://localhost:8000/HPFlights_REST/FlightOrders/"
data = {"Class" : "Business","CustomerName" :"Bhavani","DepartureDate" : "2015-10-12","FlightNumber" : "1304","NumberOfTickets": "3"}
encoded_data = urllib.urlencode(data)
'''print encoded_data
print urllib2.urlopen(url, encoded_data).read()'''
request = urllib2.Request(url, encoded_data)
print request.get_method()
request.add_data(encoded_data)
response = urllib2.urlopen(request)
Error:
Traceback (most recent call last):
File "C:/Users/kanakadurga/PycharmProjects/untitled/API.py", line 44, in <module>
createFlightOrder()
File "C:/Users/kanakadurga/PycharmProjects/untitled/API.py", line 39, in createFlightOrder
response = urllib2.urlopen(request)
File "C:\Python27\lib\urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 437, in open
response = meth(req, response)
File "C:\Python27\lib\urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 475, in error
return self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 409, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 400: Bad Request
Process finished with exit code 1
It looks like you are trying to post data to the server.
From the URL, I can make a wild guess and assume the server accepts the data in json format, probably.
If that is the case then you can do
import json
url = "http://localhost:8000/HPFlights_REST/FlightOrders/"
data = {"Class": "Business", "CustomerName": "Bhavani", "DepartureDate": "2015-10-12", "FlightNumber": "1304", "NumberOfTickets": "3"}
encoded_data = json.dumps(data)
request = urllib2.Request(url, encoded_data, {'Content-Type': 'application/json'})
f = urllib2.urlopen(req) # issue the request
response = f.read() # read the response
f.close()
... # your next operations follow
The point is that you need to encode the data correctly (json) and also set the proper content-type header in the HTTP post request, which the server probably checks.
Otherwise, the default content-type would be application/x-www-form-urlencoded, as if the data came from a form.
Related
I'm trying to make a request to the GitHub API with Python 3 urllib to create a release, but I made some mistake and it fails with an exception:
Traceback (most recent call last):
File "./a.py", line 27, in <module>
'Authorization': 'token ' + token,
File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(*args)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 422: Unprocessable Entity
GitHub however is nice, and explains why it failed on the response body as shown at: 400 vs 422 response to POST of data
So, how do I read the response body? Is there a way to prevent the exception from being raised?
I've tried to catch the exception and explore it in ipdb, which gives an object of type urllib.error.HTTPError but I couldn't find that body data there, only headers.
The script:
#!/usr/bin/env python3
import json
import os
import sys
from urllib.parse import urlencode
from urllib.request import Request, urlopen
repo = sys.argv[1]
tag = sys.argv[2]
upload_file = sys.argv[3]
token = os.environ['GITHUB_TOKEN']
url_template = 'https://{}.github.com/repos/' + repo + '/releases'
# Create.
_json = json.loads(urlopen(Request(
url_template.format('api'),
json.dumps({
'tag_namezxcvxzcv': tag,
'name': tag,
'prerelease': True,
}).encode(),
headers={
'Accept': 'application/vnd.github.v3+json',
'Authorization': 'token ' + token,
},
)).read().decode())
# This is not the tag, but rather some database integer identifier.
release_id = _json['id']
usage: Can someone give a python requests example of uploading a release asset in github?
The HTTPError has a read() method that allows you to read the response body. So in your case, you should be able to do something such as:
try:
body = urlopen(Request(
url_template.format('api'),
json.dumps({
'tag_namezxcvxzcv': tag,
'name': tag,
'prerelease': True,
}).encode(),
headers={
'Accept': 'application/vnd.github.v3+json',
'Authorization': 'token ' + token,
},
)).read().decode()
except urllib.error.HTTPError as e:
body = e.read().decode() # Read the body of the error response
_json = json.loads(body)
The docs explain in more detail how the HTTPError instance can be used as a response, and some of its other attributes.
The following curl request works perfectly when I run it through Windows' CMD.
curl -XPOST -H"Content-Type:application/json" http://my_url:8082/druid/v2/sql/ -d "{\"query\":\"SELECT DISTINCT(event) FROM programs\"}"
I am trying to replicate the same call in python 3 using urllib.requests
import urllib.request
values = {'query':'SELECT DISTINCT(event) FROM programs'}
url = 'http://my_url:8082/druid/v2/sql'
data = urllib.parse.urlencode(values).encode("utf-8")
req = urllib.request.Request(url, data)
req.add_header("Content-Type","application/json")
response = urllib.request.urlopen(req)
the_page = response.read()
However the python version is returning a Server Error
response = urllib.request.urlopen(req)
File "C:\Python\Python35\lib\urllib\request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "C:\Python\Python35\lib\urllib\request.py", line 472, in open
response = meth(req, response)
File "C:\Python\Python35\lib\urllib\request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python\Python35\lib\urllib\request.py", line 510, in error
return self._call_chain(*args)
File "C:\Python\Python35\lib\urllib\request.py", line 444, in _call_chain
result = func(*args)
File "C:\Python\Python35\lib\urllib\request.py", line 590, in
http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Server Error
Can someone tell me what I'm doing wrong please?
You need to convert your data to json first, you are sending a dictionary.
import json
data = urllib.parse.urlencode(json.dumps(values)).encode("utf-8")
Or, use the requests library:
import requests
d = {'query':'SELECT DISTINCT(event) FROM programs'}
url = 'http://my_url:8082/druid/v2/sql'
r = requests.post(url, json=d)
r.raise_for_status()
print(r.text)
After some more experimentation, I managed to make the code work by replacing urllib.parse.urlencode with json.dumps. The code now reads as follows:
import urllib.request, json
values = {'query':'SELECT DISTINCT(event) FROM programs'}
url = 'http://my_url/druid/v2/sql'
data = json.dumps(values).encode("utf-8")
req = urllib.request.Request(url, data)
req.add_header("Content-Type","application/json")
response = urllib.request.urlopen(req)
the_page = response.read()
I'm trying to download the HTML of a page (http://www.guangxindai.com in this case) but I'm getting back an error 403. Here is my code:
import urllib.request
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
f = opener.open("http://www.guangxindai.com")
f.read()
but I get error response.
Traceback (most recent call last):
File "<pyshell#7>", line 1, in <module>
f = opener.open("http://www.guangxindai.com")
File "C:\Python33\lib\urllib\request.py", line 475, in open
response = meth(req, response)
File "C:\Python33\lib\urllib\request.py", line 587, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python33\lib\urllib\request.py", line 513, in error
return self._call_chain(*args)
File "C:\Python33\lib\urllib\request.py", line 447, in _call_chain
result = func(*args)
File "C:\Python33\lib\urllib\request.py", line 595, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
I have tried different request headers, but still can not get correct response. I can view the web through browser. It seems strange for me. I guess the web use some method to block web spider. Does anyone know what is happening? How can I get the HTML of page correctly?
I was having the same problem that you and I found the answer in this link.
The answer provided by Stefano Sanfilippo is quite simple and worked for me:
from urllib.request import Request, urlopen
url_request = Request("http://www.guangxindai.com",
headers={"User-Agent": "Mozilla/5.0"})
webpage = urlopen(url_request).read()
If your aim is to read the html of the page you can use the following code. It worked for me on Python 2.7
import urllib
f = urllib.urlopen("http://www.guangxindai.com")
f.read()
I am working on an automation script (that I am using to automate the process of conversion of some videos). In this script after video conversion, I am calling my web service to update the clip status in database and sending the web service a list of clips in POST request. But the problem is this request is failing and causing 500 internal server error on server side.
Here is the code I am using to call the web service with sample data I am trying with:
post_body = {
'clips': [
{
'clip_id': 17555,
'db_url': '/720p/14555.mp4'
}
]
}
params = urlencode(post_body)
url = str(self.update_url)
req = urllib2.Request(url, params)
response = urllib2.urlopen(req)
res = response.read()
print res
And here is the code of my web service:
def update_conversion_clips(request):
print "Web service is called"
try:
clips = request.POST.get('clips', None)
print clips
return HttpResponse(True)
except:
return HttpResponse(False)
Even first print statement is not executing.
Here is the error stack trace on application side:
Traceback (most recent call last):
File "conversion_script.py", line 48, in <module>
conversion_script.run()
File "conversion_script.py", line 44, in run
self.clips.update_clips_info(None)
File "/home/abc/video_experiments/conversion/clips_manager.py", line 59, in update_clips_info
response = urllib2.urlopen(req)
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 406, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 444, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR
and this is error on server side:
[20/Feb/2014 04:13:15] "POST /update_conversion_clips HTTP/1.1" 500 68733
According to my research this is happening due to multilevel dict that I am sending in POST. But I could not find any solution to resolve it.
New code now sending data as json (still does not work):
values = dict()
values['clips'] = [
{
'clip_id': 17555,
'db_url': '/720p/14555.mp4'
}
]
req = urllib2.Request(self.update_url)
req.add_header('Content-Type', 'application/json')
response = urllib2.urlopen(req, json.dumps(values))
res = response.read()
print res
and on server side:
try:
data = json.loads(request.body)
clips = data['clips']
except:
print "Exception occured!"
HttpResponse(True)
urlencode isn't really a good format for this data. A much better one would be JSON.
req = urllib2.Request(self.update_url)
req.add_header('Content-Type', 'application/json')
response = urllib2.urlopen(req, json.dumps(data))
print response.read()
(you could make this part a lot simpler by using the third-party requests library).
And in the server:
clips = json.loads(request.body)
I am fairly inexperienced with user authentication especially through restful apis. I am trying to use python to log in with a user that is set up in parse.com. The following is the code I have:
API_LOGIN_ROOT = 'https://api.parse.com/1/login'
params = {'username':username,'password':password}
encodedParams = urllib.urlencode(params)
url = API_LOGIN_ROOT + "?" + encodedParams
request = urllib2.Request(url)
request.add_header('Content-type', 'application/x-www-form-urlencoded')
# we could use urllib2's authentication system, but it seems like overkill for this
auth_header = "Basic %s" % base64.b64encode('%s:%s' % (APPLICATION_ID, MASTER_KEY))
request.add_header('Authorization', auth_header)
request.add_header('X-Parse-Application-Id', APPLICATION_ID)
request.add_header('X-Parse-REST-API-Key', MASTER_KEY)
request.get_method = lambda: http_verb
# TODO: add error handling for server response
response = urllib2.urlopen(request)
#response_body = response.read()
#response_dict = json.loads(response_body)
This is a modification of an open source library used to access the parse rest interface.
I get the following error:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__
handler.post(*groups)
File "/Users/nazbot/src/PantryPal_AppEngine/fridgepal.py", line 464, in post
url = user.login()
File "/Users/nazbot/src/PantryPal_AppEngine/fridgepal.py", line 313, in login
url = self._executeCall(self.username, self.password, 'GET', data)
File "/Users/nazbot/src/PantryPal_AppEngine/fridgepal.py", line 292, in _executeCall
response = urllib2.urlopen(request)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 438, in error
return self._call_chain(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 521, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
Can someone point me to where I am screwing up? I'm not quite sure why I'm getting a 404 instead of an access denied or some other issue.
Make sure the "User" class was created on Parse.com as a special user class. When you are adding the class, make sure to change the Class Type to "User" instead of "Custom". A little user head icon will show up next to the class name on the left hand side.
This stumped me for a long time until Matt from the Parse team showed me the problem.
Please change: API_LOGIN_ROOT = 'https://api.parse.com/1/login' to the following: API_LOGIN_ROOT = 'https://api.parse.com/1/login**/**'
I had the same problem using PHP, adding the / at the end fixed the 404 error.