I've spent a total of 30 minutes in python lol, so take that into consideration when you answer lol:
I'm trying to send an HTTP POST request with a body and reading the response. I'm using Python 3.6.5 on Windows 10. This is what I have so far:
import http.client
import xml.dom.minidom
HOST = "www.mysite.com"
API_URL = "/service"
def do_request(xml_location):
request = open(xml_location, "r").read()
webservice = http.client.HTTPConnection(HOST)
webservice.request("POST", API_URL)
webservice.putheader("Host", HOST)
webservice.putheader("User-Agent", "Python Post")
webservice.putheader("Content-type", "text/xml; charset=\"UTF-8\"")
webservice.putheader("Content-length", "%d" % len(request))
webservice.endheaders()
webservice.send(request)
statuscode, statusmessage, header = webservice.getreply()
result = webservice.getfile().read()
resultxml = xml.dom.minidom.parseString(result)
print (statuscode, statusmessage, header)
print (resultxml.toprettyxml())
with open("output-%s" % xml_location, "w") as xmlfile:
xmlfile.write(resultxml.toprettyxml())
do_request("test.xml")
test.xml contains the XML request. When I run, I get an error:
Traceback (most recent call last):
File "C:\Users\xxx\Documents\test.py", line 33, in <module>
do_request("test.xml")
File "C:\Users\xxx\Documents\test.py", line 14, in do_request
webservice.putheader("Host", HOST)
File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\http\client.py", line 1201, in putheader
raise CannotSendHeader()
http.client.CannotSendHeader
Your problem is that you mixed up the request and putrequest methods. (Not surprisingly, given the brevity and sparsity of the documentation… most modules in Python are documented a lot better than this, so don't let that worry you about the future.)
The request method is a convenience function that adds the request line, all the headers, and the data all in one go. After you've done that, it's way too late to add a header, hence the error message.
So, you can fix it either way.
(1) Change it to use putrequest. I realize there's no example using putrequest or putheader anywhere in the docs, but it looks like this:
webservice.putrequest("POST", API_URL)
webservice.putheader("Host", HOST)
webservice.putheader("User-Agent", "Python Post")
webservice.putheader("Content-type", "text/xml; charset=\"UTF-8\"")
webservice.putheader("Content-length", "%d" % len(request))
webservice.endheaders()
webservice.send(request)
(2) Change it to use request. This is what all the examples in the docs do; you just need to build up a dict of headers to pass to it:
headers = {
"Host": HOST,
"User-Agent": "Python Post",
"Content-type", "text/xml; charset=\"UTF-8\"",
"Content-length", "%d" % len(request)
}
webservice.request("POST", API_URL, headers=headers, body=request)
(3) Read this at the top of the docs:
This module defines classes which implement the client side of the HTTP and HTTPS protocols. It is normally not used directly — the module urllib.request uses it to handle URLs that use HTTP and HTTPS.
See also The Requests package is recommended for a higher-level HTTP client interface.
For most real-life cases, you want to use requests if you can use a third-party library, and urllib.request if you can't. They're both simpler, and better documented.
Related
I am communicating with an API using HTTP.client in Python 3.6.2.
In order to upload a file it requires a three stage process.
I have managed to talk successfully using POST methods and the server returns data as I expect.
However, the stage that requires the actual file to be uploaded is a PUT method - and I cannot figure out how to syntax the code to include a pointer to the actual file on my storage - the file is an mp4 video file.
Here is a snippet of the code with my noob annotations :)
#define connection as HTTPS and define URL
uploadstep2 = http.client.HTTPSConnection("grabyo-prod.s3-accelerate.amazonaws.com")
#define headers
headers = {
'accept': "application/json",
'content-type': "application/x-www-form-urlencoded"
}
#define the structure of the request and send it.
#Here it is a PUT request to the unique URL as defined above with the correct file and headers.
uploadstep2.request("PUT", myUniqueUploadUrl, body="C:\Test.mp4", headers=headers)
#get the response from the server
uploadstep2response = uploadstep2.getresponse()
#read the data from the response and put to a usable variable
step2responsedata = uploadstep2response.read()
The response I am getting back at this stage is an
"Error 400 Bad Request - Could not obtain the file information."
I am certain this relates to the body="C:\Test.mp4" section of the code.
Can you please advise how I can correctly reference a file within the PUT method?
Thanks in advance
uploadstep2.request("PUT", myUniqueUploadUrl, body="C:\Test.mp4", headers=headers)
will put the actual string "C:\Test.mp4" in the body of your request, not the content of the file named "C:\Test.mp4" as you expect.
You need to open the file, read it's content then pass it as body. Or to stream it, but AFAIK http.client does not support that, and since your file seems to be a video, it is potentially huge and will use plenty of RAM for no good reason.
My suggestion would be to use requests, which is a way better lib to do this kind of things:
import requests
with open(r'C:\Test.mp4'), 'rb') as finput:
response = requests.put('https://grabyo-prod.s3-accelerate.amazonaws.com/youruploadpath', data=finput)
print(response.json())
I do not know if it is useful for you, but you can try to send a POST request with requests module :
import requests
url = ""
data = {'title':'metadata','timeDuration':120}
mp3_f = open('/path/your_file.mp3', 'rb')
files = {'messageFile': mp3_f}
req = requests.post(url, files=files, json=data)
print (req.status_code)
print (req.content)
Hope it helps .
I have some python code that looks like the following:
import urllib3
http = urllib3.PoolManager(cert_reqs='CERT_NONE')
...
full_url = 'https://[%s]:%d%s%s' % \
(address, port, base_uri, relative_uri)
kwargs = {
'headers': {
'Host': '%s:%d' % (hostname, port)
}
}
if data is not None:
kwargs['body'] = json.dumps(data, indent=2, sort_keys=True)
# Directly use request_encode_url instead of request because requests
# will try to encode the body as 'multipart/form-data'.
response = http.request_encode_url('POST', full_url, **kwargs)
log.debug('Received response: HTTP status %d. Body: %s' %
(response.status, repr(response.data)))
I have a log line that prints once prior to the code that issues the request, and the log.debug('Received...') line prints once. However, on the server side, I occasionally see two requests (they are both the same POST request that is sent by this code block), around 1-5 seconds apart. In such instances, the order of events is as follows:
One request sent from python client
First request received
Second request received
First response sent with status 200 and an http entity indicating success
Second response sent with status 200 and http entity indicating failure
Python client receives the second reponse
I tried to reproduce it reliably by sleeping in the server (guessing that there might be a timeout that causes a retry), but was unsuccessful. I believe the duplication is unlikely to be occurring on the server because it's just a basic Scala Spray server and haven't seen this with other clients. Looking at the source code for PoolManager, I can't find anywhere where retries would be included. There is a mechanism for retries specified with an optional parameter, but this optional parameter is not being used in the code above.
Does anyone have any ideas where this extra request might be coming from?
EDIT: #shazow gave a pointer about retries having a default of 3, but I changed the code as suggested and got the following error:
Traceback (most recent call last):
File "my_file.py", line 23, in <module>
response = http.request_encode_url('GET', full_url, **kwargs)
File "/usr/lib/python2.7/dist-packages/urllib3/request.py", line 88, in request_encode_url
return self.urlopen(method, url, **urlopen_kw)
File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 145, in urlopen
conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 119, in connection_from_host
pool = self._new_pool(scheme, host, port)
File "/usr/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 86, in _new_pool
return pool_cls(host, port, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'retries'`
Edit #2: The following change to kwargs seems to work for me:
import urllib3
http = urllib3.PoolManager(cert_reqs='CERT_NONE')
...
full_url = 'https://[%s]:%d%s%s' % \
(address, port, base_uri, relative_uri)
kwargs = {
'headers': {
'Host': '%s:%d' % (hostname, port)
},
'retries': 0
}
if data is not None:
kwargs['body'] = json.dumps(data, indent=2, sort_keys=True)
# Directly use request_encode_url instead of request because requests
# will try to encode the body as 'multipart/form-data'.
response = http.request_encode_url('POST', full_url, **kwargs)
log.debug('Received response: HTTP status %d. Body: %s' %
(response.status, repr(response.data)))
urllib3 has a default retries configuration, which is the equivalent to Retry(3). To disable retries outright, you'll need to pass retries=False either when constructing the pool or making a request.
Something like this should work, for example:
import urllib3
http = urllib3.PoolManager(cert_reqs='CERT_NONE', retries=False)
...
The default retries setting (as defined here) could definitely be better documented, I would appreciate your contribution if you feel up for it. :)
I'm currently working on a automated way to interface with a database website that has RESTful webservices installed. I am having issues with figure out the proper formatting of how to properly send the requests listed in the following site using python.
https://neesws.neeshub.org:9443/nees.html
Particular example is this:
POST https://neesws.neeshub.org:9443/REST/Project/731/Experiment/1706/Organization
<Organization id="167"/>
The biggest problem is that I do not know where to put the XML formatted part of the above. I want to send the above as a python HTTPS request and so far I've been trying something of the following structure.
>>>import httplib
>>>conn = httplib.HTTPSConnection("neesws.neeshub.org:9443")
>>>conn.request("POST", "/REST/Project/731/Experiment/1706/Organization")
>>>conn.send('<Organization id="167"/>')
But this appears to be completely wrong. I've never actually done python when it comes to webservices interfaces so my primary question is how exactly am I supposed to use httplib to send the POST Request, particularly the XML formatted part of it? Any help is appreciated.
You need to set some request headers before sending data. For example, content-type to 'text/xml'. Checkout the few examples,
Post-XML-Python-1
Which has this code as example:
import sys, httplib
HOST = www.example.com
API_URL = /your/api/url
def do_request(xml_location):
"""HTTP XML Post requeste"""
request = open(xml_location,"r").read()
webservice = httplib.HTTP(HOST)
webservice.putrequest("POST", API_URL)
webservice.putheader("Host", HOST)
webservice.putheader("User-Agent","Python post")
webservice.putheader("Content-type", "text/xml; charset=\"UTF-8\"")
webservice.putheader("Content-length", "%d" % len(request))
webservice.endheaders()
webservice.send(request)
statuscode, statusmessage, header = webservice.getreply()
result = webservice.getfile().read()
print statuscode, statusmessage, header
print result
do_request("myfile.xml")
Post-XML-Python-2
You may get some idea.
I'm trying to learn how urllib2 works and how it encapsulates its various components before sending out an actual request or response.
So far I have:
theurl = "www.example.com"
That obviously specifies the URL to look at.
req = urllib2.Request(theurl)
Don't know what this does, hence the question.
handle = urllib2.urlopen(req)
This one gets the page and does all the requests and responses required.
So my question is, what does urllib2.Request actually do?
To try and look at it to get an idea I tried
print req
and just got
<urllib2.Request instance at 0x123456789>
I also tried
print req.read()
and got:
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/lib64/python2.4/urllib2.py, line 207, in `__`getattr`__`
raise AttributeError, attr
AttributeError: read
So I'm obviously doing something wrong. If anyone can help in one of both my questions that would be great.
The class "Request" you're asking about:
http://docs.python.org/library/urllib2.html#urllib2.Request
class urllib2.Request(url[, data][,
headers][, origin_req_host][,
unverifiable])
This class is an abstraction of a URL
request.
The function you actually want to make a request (which can accept a Request object or wrap one around a URL string you provice) constructing a Request object): http://docs.python.org/library/urllib2.html#urllib2.urlopen
urllib2.urlopen(url[, data][,timeout])
Open the URL url, which can be either a string or a Request object.
Example:
theurl = "www.example.com"
try:
resp = urllib2.urlopen(theurl)
print resp.read()
except IOError as e:
print "Error: ", e
Example 2 (with Request):
theurl = "www.example.com"
try:
req = urllib2.Request(theurl)
print req.get_full_url()
print req.get_method()
print dir(req) # list lots of other stuff in Request
resp = urllib2.urlopen(req)
print resp.read()
except IOError as e:
print "Error: ", e
urllib2.Request() looks like a function call, but isn't - it's an object constructor. It creates an object of type Request from the urllib2 module, documented here.
As such, it probably doesn't do anything except initialise itself. You can verify this by looking at the source code, which should be in your Python installation's lib directory (urllib2.py, at least in Python 2.x).
If you want to have the constructed URL in the Request object use :
print(urllib2.Request.get_full_url())
I'm newbie for python, I'm having task so I need to scan wifi and send the data to the server, the below is the format which i have to send, this work fine when enter manually in browser url text box,
http://223.56.124.58:8080/ppod-web/ProcessRawData?data={"userId":"2220081127-14","timestamp":"2010-04-12 10:54:24","wifi":{"ssid":"guest","rssi":"80"}}
here is my code:
import httplib
import urllib
params = urllib.urlencode('{\"userId\":\"20081127-14\",\"timestamp\":\"2010-04-12 10:54:24\",\"wifi\":{\"ssid\":\"guest\",\"rssi\":\"80\"}}')
headers = {"Content-type":"application/x-www-form-urlencoded","Accept":"text/plain"}
conn = httplib.HTTPConnection("http://223.56.124.58:8080")
conn.request("POST","ppod-web/ProcessRawData?data=",params,headers)
response = conn.getresponse()
print response.status
print "-----"
print response.reason
data = response.read()
print data
conn.close()
thanks
Most likely, the issue with the script you posted in the question is you cannot directly do:
conn=httplib.HTTPConnection("http://223.56.124.58:8080/wireless")
The exception is triggered in getaddrinfo(), which calls the C function getaddrinfo() which returns EAI_NONAME:
The node or service is not known; or both node and service are NULL; or AI_NUMERICSERV was specified in hints.ai_flags and service was not a numeric port-number string."
There obviously is a problem with the parameters passed to getaddrinfo, and most likely you are trying to get information for the "223.56.124.58:8080/wireless" host. Ooops!
Indeed, you cannot directly connect to an URL address. As the documentation clearly states and shows, you connect to the server:
conn = httplib.HTTPConnection("223.56.124.58", 8080)
Then you can do:
conn.request("POST", "wireless", params, headers)
What about the script you are actually using?
conn.request("POST","http://202.45.139.58:8080/ppod-web",params,headers)
Even if the connection was correctly formed, that would have you POSTing to http://202.45.139.58:8080/http://202.45.139.58:8080/ppod-web. What you really want probably is:
conn = httplib.HTTPConnection("202.45.139.58", 8080)
conn.request("POST", "ppod-web", params, headers)
The error is shown for this line because most likely HTTPConnection is a lazy object and only attempts to actually connect to the server when you call request().
After you're done fixing the above, you'll need to fix params.
>>> urllib.urlencode({"wifi":{"ssid":"guest","rssi","80"}})
SyntaxError: invalid syntax
>>> urllib.urlencode({"wifi":{"ssid":"guest","rssi":"80"}})
'wifi=%7B%27rssi%27%3A+%2780%27%2C+%27ssid%27%3A+%27guest%27%7D'
To get what you think you want to get, you should do:
>>> urllib.urlencode({"data": {"wifi":{"ssid":"guest","rssi":"80"}}})
'data=%7B%27wifi%27%3A+%7B%27rssi%27%3A+%2780%27%2C+%27ssid%27%3A+%27guest%27%7D%7D'
Instead of:
conn = httplib.HTTPConnection("http://223.56.124.58:8080/wireless")
conn.request("POST", "data", params, headers)
try:
conn = httplib.HTTPConnection("223.56.124.58", port=8080)
conn.request("POST", "/wireless", params, headers)
Not sure if it will resolve all your problems, but at least your code will conform to the method/constructor signatures.
The traceback doesn't come from the same code you pasted.
On the error traceback there's a line:
conn.request("POST","http://202.45.139.58:8080/ppod-web",params,headers)
It is the line 9 of http.py however it is not on the code you pasted.
Please paste the actual code.