I am working on an automation script (that I am using to automate the process of conversion of some videos). In this script after video conversion, I am calling my web service to update the clip status in database and sending the web service a list of clips in POST request. But the problem is this request is failing and causing 500 internal server error on server side.
Here is the code I am using to call the web service with sample data I am trying with:
post_body = {
'clips': [
{
'clip_id': 17555,
'db_url': '/720p/14555.mp4'
}
]
}
params = urlencode(post_body)
url = str(self.update_url)
req = urllib2.Request(url, params)
response = urllib2.urlopen(req)
res = response.read()
print res
And here is the code of my web service:
def update_conversion_clips(request):
print "Web service is called"
try:
clips = request.POST.get('clips', None)
print clips
return HttpResponse(True)
except:
return HttpResponse(False)
Even first print statement is not executing.
Here is the error stack trace on application side:
Traceback (most recent call last):
File "conversion_script.py", line 48, in <module>
conversion_script.run()
File "conversion_script.py", line 44, in run
self.clips.update_clips_info(None)
File "/home/abc/video_experiments/conversion/clips_manager.py", line 59, in update_clips_info
response = urllib2.urlopen(req)
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 406, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 444, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 500: INTERNAL SERVER ERROR
and this is error on server side:
[20/Feb/2014 04:13:15] "POST /update_conversion_clips HTTP/1.1" 500 68733
According to my research this is happening due to multilevel dict that I am sending in POST. But I could not find any solution to resolve it.
New code now sending data as json (still does not work):
values = dict()
values['clips'] = [
{
'clip_id': 17555,
'db_url': '/720p/14555.mp4'
}
]
req = urllib2.Request(self.update_url)
req.add_header('Content-Type', 'application/json')
response = urllib2.urlopen(req, json.dumps(values))
res = response.read()
print res
and on server side:
try:
data = json.loads(request.body)
clips = data['clips']
except:
print "Exception occured!"
HttpResponse(True)
urlencode isn't really a good format for this data. A much better one would be JSON.
req = urllib2.Request(self.update_url)
req.add_header('Content-Type', 'application/json')
response = urllib2.urlopen(req, json.dumps(data))
print response.read()
(you could make this part a lot simpler by using the third-party requests library).
And in the server:
clips = json.loads(request.body)
Related
I have used requests library and I know how to work with it, but I need to work with standard library only, so I would appreciate if you don't encourage me to use requests instead.
I made a flask server that handles POST requests and then from a different script I call urllib to make POST calls to the flask server. I need to send a raw json in body just like we do in Postman.
Flask Server
from flask import Flask, request
app = Flask(__name__)
#app.route('/random', methods=['POST'])
def random():
if request.method == 'POST':
if request.headers.get('Authorization') and request.headers.get('Content-Type') == 'application/json':
print(request.get_json())
return "Success"
else:
print(request.get_json())
return "Bad request"
app.run(host='0.0.0.0', port=5000, debug=True)
Urllib Client (saved as test.py) -
import urllib.request
import urllib.parse
d = {"spam": 1, "eggs": 2, "bacon": 0}
data = urllib.parse.urlencode(d)
data = data.encode()
req = urllib.request.Request("http://localhost:5000/random", data)
req.add_header('Content-Type', 'application/json')
req.add_header('Authorization', 12345)
with urllib.request.urlopen(req) as f:
print(f.read().decode('utf-8'))
With only Authorization header I get Bad Request as output and the json is None on the flask server side as expected.
With ONLY Content-Type header OR both the headers I get this error -
Traceback (most recent call last):
File "test.py", line 9, in <module>
with urllib.request.urlopen(req) as f:
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 570, in error
return self._call_chain(*args)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\ProgramData\Anaconda3\lib\urllib\request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: BAD REQUEST
The whole thing is simple enough, but I am not able to understand why is this happening and the error message doesn't help much either.
The server is failing in request.get_json(). It's only happening when the client sends both headers because that's when it reaches this line.
To fix it, change the client to send the data as JSON:
import json # <-- Import json
import urllib.request
import urllib.parse
d = {"spam": 1, "eggs": 2, "bacon": 0}
data = json.dumps(d) # <-- Dump the dictionary as JSON
data = data.encode()
req = urllib.request.Request("http://localhost:5000/random", data)
req.add_header('Content-Type', 'application/json')
req.add_header('Authorization', 12345)
with urllib.request.urlopen(req) as f:
print(f.read().decode('utf-8'))
I hope this helps
The following curl request works perfectly when I run it through Windows' CMD.
curl -XPOST -H"Content-Type:application/json" http://my_url:8082/druid/v2/sql/ -d "{\"query\":\"SELECT DISTINCT(event) FROM programs\"}"
I am trying to replicate the same call in python 3 using urllib.requests
import urllib.request
values = {'query':'SELECT DISTINCT(event) FROM programs'}
url = 'http://my_url:8082/druid/v2/sql'
data = urllib.parse.urlencode(values).encode("utf-8")
req = urllib.request.Request(url, data)
req.add_header("Content-Type","application/json")
response = urllib.request.urlopen(req)
the_page = response.read()
However the python version is returning a Server Error
response = urllib.request.urlopen(req)
File "C:\Python\Python35\lib\urllib\request.py", line 163, in urlopen
return opener.open(url, data, timeout)
File "C:\Python\Python35\lib\urllib\request.py", line 472, in open
response = meth(req, response)
File "C:\Python\Python35\lib\urllib\request.py", line 582, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python\Python35\lib\urllib\request.py", line 510, in error
return self._call_chain(*args)
File "C:\Python\Python35\lib\urllib\request.py", line 444, in _call_chain
result = func(*args)
File "C:\Python\Python35\lib\urllib\request.py", line 590, in
http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Server Error
Can someone tell me what I'm doing wrong please?
You need to convert your data to json first, you are sending a dictionary.
import json
data = urllib.parse.urlencode(json.dumps(values)).encode("utf-8")
Or, use the requests library:
import requests
d = {'query':'SELECT DISTINCT(event) FROM programs'}
url = 'http://my_url:8082/druid/v2/sql'
r = requests.post(url, json=d)
r.raise_for_status()
print(r.text)
After some more experimentation, I managed to make the code work by replacing urllib.parse.urlencode with json.dumps. The code now reads as follows:
import urllib.request, json
values = {'query':'SELECT DISTINCT(event) FROM programs'}
url = 'http://my_url/druid/v2/sql'
data = json.dumps(values).encode("utf-8")
req = urllib.request.Request(url, data)
req.add_header("Content-Type","application/json")
response = urllib.request.urlopen(req)
the_page = response.read()
I am learning python API testing using urllib2 module.I tried to execute the code.but throwing the following msg.Can anybody help me.Thanks in advance.
code:
url = "http://localhost:8000/HPFlights_REST/FlightOrders/"
data = {"Class" : "Business","CustomerName" :"Bhavani","DepartureDate" : "2015-10-12","FlightNumber" : "1304","NumberOfTickets": "3"}
encoded_data = urllib.urlencode(data)
'''print encoded_data
print urllib2.urlopen(url, encoded_data).read()'''
request = urllib2.Request(url, encoded_data)
print request.get_method()
request.add_data(encoded_data)
response = urllib2.urlopen(request)
Error:
Traceback (most recent call last):
File "C:/Users/kanakadurga/PycharmProjects/untitled/API.py", line 44, in <module>
createFlightOrder()
File "C:/Users/kanakadurga/PycharmProjects/untitled/API.py", line 39, in createFlightOrder
response = urllib2.urlopen(request)
File "C:\Python27\lib\urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 437, in open
response = meth(req, response)
File "C:\Python27\lib\urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python27\lib\urllib2.py", line 475, in error
return self._call_chain(*args)
File "C:\Python27\lib\urllib2.py", line 409, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 400: Bad Request
Process finished with exit code 1
It looks like you are trying to post data to the server.
From the URL, I can make a wild guess and assume the server accepts the data in json format, probably.
If that is the case then you can do
import json
url = "http://localhost:8000/HPFlights_REST/FlightOrders/"
data = {"Class": "Business", "CustomerName": "Bhavani", "DepartureDate": "2015-10-12", "FlightNumber": "1304", "NumberOfTickets": "3"}
encoded_data = json.dumps(data)
request = urllib2.Request(url, encoded_data, {'Content-Type': 'application/json'})
f = urllib2.urlopen(req) # issue the request
response = f.read() # read the response
f.close()
... # your next operations follow
The point is that you need to encode the data correctly (json) and also set the proper content-type header in the HTTP post request, which the server probably checks.
Otherwise, the default content-type would be application/x-www-form-urlencoded, as if the data came from a form.
I am trying to connect to a REST resource and retrieve the data using Python script (Python 3.2.3). When I run the script I am getting error as HTTP Error 401: Unauthorized. Please note that I am able to access the given REST resource using REST client using Basic Authentication. In the REST Client I have specified the hostname, user and password details (realm is not required).
Below is the code and complete error. Your help is very much appreciated.
Code:
import urllib.request
# set up authentication info
auth_handler = urllib.request.HTTPBasicAuthHandler()
auth_handler.add_password(realm=None,
uri=r'http://hostname/',
user='administrator',
passwd='administrator')
opener = urllib.request.build_opener(auth_handler)
urllib.request.install_opener(opener)
res = opener.open(r'http://hostname:9004/apollo-api/nodes')
nodes = res.read()
Error
Traceback (most recent call last):
File "C:\Python32\scripts\get-nodes.py", line 12, in <module>
res = opener.open(r'http://tolowa.wysdm.lab.emc.com:9004/apollo-api/nodes')
File "C:\Python32\lib\urllib\request.py", line 375, in open
response = meth(req, response)
File "C:\Python32\lib\urllib\request.py", line 487, in http_response
'http', request, response, code, msg, hdrs)
File "C:\Python32\lib\urllib\request.py", line 413, in error
return self._call_chain(*args)
File "C:\Python32\lib\urllib\request.py", line 347, in _call_chain
result = func(*args)
File "C:\Python32\lib\urllib\request.py", line 495, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 401: Unauthorized
Try to give the correct realm name. You can find this out for example when opening the page in a browser - the password prompt should display the name.
You can also read the realm by catching the exception that was raised:
import urllib.error
import urllib.request
# set up authentication info
auth_handler = urllib.request.HTTPBasicAuthHandler()
auth_handler.add_password(realm=None,
uri=r'http://hostname/',
user='administrator',
passwd='administrator')
opener = urllib.request.build_opener(auth_handler)
urllib.request.install_opener(opener)
try:
res = opener.open(r'http://hostname:9004/apollo-api/nodes')
nodes = res.read()
except urllib.error.HTTPError as e:
print(e.headers['www-authenticate'])
You should get the following output:
Basic realm="The realm you are after"
Read the realm from above and set it in your add_password method and it should be good to go.
I am fairly inexperienced with user authentication especially through restful apis. I am trying to use python to log in with a user that is set up in parse.com. The following is the code I have:
API_LOGIN_ROOT = 'https://api.parse.com/1/login'
params = {'username':username,'password':password}
encodedParams = urllib.urlencode(params)
url = API_LOGIN_ROOT + "?" + encodedParams
request = urllib2.Request(url)
request.add_header('Content-type', 'application/x-www-form-urlencoded')
# we could use urllib2's authentication system, but it seems like overkill for this
auth_header = "Basic %s" % base64.b64encode('%s:%s' % (APPLICATION_ID, MASTER_KEY))
request.add_header('Authorization', auth_header)
request.add_header('X-Parse-Application-Id', APPLICATION_ID)
request.add_header('X-Parse-REST-API-Key', MASTER_KEY)
request.get_method = lambda: http_verb
# TODO: add error handling for server response
response = urllib2.urlopen(request)
#response_body = response.read()
#response_dict = json.loads(response_body)
This is a modification of an open source library used to access the parse rest interface.
I get the following error:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/webapp/_webapp25.py", line 703, in __call__
handler.post(*groups)
File "/Users/nazbot/src/PantryPal_AppEngine/fridgepal.py", line 464, in post
url = user.login()
File "/Users/nazbot/src/PantryPal_AppEngine/fridgepal.py", line 313, in login
url = self._executeCall(self.username, self.password, 'GET', data)
File "/Users/nazbot/src/PantryPal_AppEngine/fridgepal.py", line 292, in _executeCall
response = urllib2.urlopen(request)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open
response = meth(req, response)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 438, in error
return self._call_chain(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 521, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
Can someone point me to where I am screwing up? I'm not quite sure why I'm getting a 404 instead of an access denied or some other issue.
Make sure the "User" class was created on Parse.com as a special user class. When you are adding the class, make sure to change the Class Type to "User" instead of "Custom". A little user head icon will show up next to the class name on the left hand side.
This stumped me for a long time until Matt from the Parse team showed me the problem.
Please change: API_LOGIN_ROOT = 'https://api.parse.com/1/login' to the following: API_LOGIN_ROOT = 'https://api.parse.com/1/login**/**'
I had the same problem using PHP, adding the / at the end fixed the 404 error.