There's a lot of stuff out there on urllib2 and POST calls, but I'm stuck on a problem.
I'm trying to do a simple POST call to a service:
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
content = urllib2.urlopen(url=url, data=data).read()
print content
I can see the server logs and it says that I'm doing GET calls, when I'm sending the data
argument to urlopen.
The library is raising an 404 error (not found), which is correct for a GET call, POST calls are processed well (I'm also trying with a POST within a HTML form).
Do it in stages, and modify the object, like this:
# make a string with the request type in it:
method = "POST"
# create a handler. you can specify different handlers here (file uploads etc)
# but we go for the default
handler = urllib2.HTTPHandler()
# create an openerdirector instance
opener = urllib2.build_opener(handler)
# build a request
data = urllib.urlencode(dictionary_of_POST_fields_or_None)
request = urllib2.Request(url, data=data)
# add any other information you want
request.add_header("Content-Type",'application/json')
# overload the get method function with a small anonymous function...
request.get_method = lambda: method
# try it; don't forget to catch the result
try:
connection = opener.open(request)
except urllib2.HTTPError,e:
connection = e
# check. Substitute with appropriate HTTP code.
if connection.code == 200:
data = connection.read()
else:
# handle the error case. connection.read() will still contain data
# if any was returned, but it probably won't be of any use
This way allows you to extend to making PUT, DELETE, HEAD and OPTIONS requests too, simply by substituting the value of method or even wrapping it up in a function. Depending on what you're trying to do, you may also need a different HTTP handler, e.g. for multi file upload.
This may have been answered before: Python URLLib / URLLib2 POST.
Your server is likely performing a 302 redirect from http://myserver/post_service to http://myserver/post_service/. When the 302 redirect is performed, the request changes from POST to GET (see Issue 1401). Try changing url to http://myserver/post_service/.
Have a read of the urllib Missing Manual. Pulled from there is the following simple example of a POST request.
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe', 'age' : '10'})
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
print response.read()
As suggested by #Michael Kent do consider requests, it's great.
EDIT: This said, I do not know why passing data to urlopen() does not result in a POST request; It should. I suspect your server is redirecting, or misbehaving.
The requests module may ease your pain.
url = 'http://myserver/post_service'
data = dict(name='joe', age='10')
r = requests.post(url, data=data, allow_redirects=True)
print r.content
it should be sending a POST if you provide a data parameter (like you are doing):
from the docs:
"the HTTP request will be a POST instead of a GET when the data parameter is provided"
so.. add some debug output to see what's up from the client side.
you can modify your code to this and try again:
import urllib
import urllib2
url = 'http://myserver/post_service'
opener = urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1))
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
content = opener.open(url, data=data).read()
Try this instead:
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
req = urllib2.Request(url=url,data=data)
content = urllib2.urlopen(req).read()
print content
url="https://myserver/post_service"
data["name"] = "joe"
data["age"] = "20"
data_encoded = urllib2.urlencode(data)
print urllib2.urlopen(url + "?" + data_encoded).read()
May be this can help
Related
I need to print all requests from specific Postman collection. I have this code:
import requests
# Set up Postman API endpoint and authorization
postman_api_endpoint = "https://api.getpostman.com/collections"
postman_api_key = "PMAK-63b6bf724ebf902ad13d4bf2-e683c12d426716552861acda**********"
headers = {"X-Api-Key": postman_api_key}
# Get all requests from Postman collection
collection_id = "25184041-c1537769-f598-4c0e-b8ae-8cd185a79c03"
response = requests.get(f"{postman_api_endpoint}/{collection_id}/items", headers)
if response.status_code != 200:
print("Error retrieving collection:", response.text)
else:
# Print all requests
requests_data = response.json()["items"]
for request_data in requests_data:
request_method = request_data["request"]["method"]
request_url = request_data["request"]["url"]
request_headers = request_data["request"]["header"]
request_body = request_data["request"]["body"]["raw"] \
if request_data["request"]["body"]["mode"] == "raw" else ""
print(f"{request_method} {request_url}")
print("Headers:")
for header in request_headers:
print(f"{header['key']}: {header['value']}")
print("Body:")
print(request_body)
I received an error while I try to call response.text and have such massage:
Error retrieving collection: {"error":{"name":"notFound","message":"Requested resource not found"}}
Which means that I have 404 error. I have several assumptions what I did wrong:
I entered incorrect api key(But I checked several times and regenerated it twice)
I entered incorrect collection id, but in the screen below you can see where I took it and it is correct
And as I think the most likely variant I wrote incorrect request where I put my key and my collection id(I din't find any example how such requests should be like)
And of course I have requests in my collection, so error can not be because the collection is empty
Please give me some advice how I can fix this error. Thank you!
The answer is actually is really simple. I didn't know that I need to push button save request in Postman. I think if I create request in collection it will automatically save it. But I didn't, so I just saved all requests manually and finally receive correct response.
If I make the request
api-key = 'asdfklhsdfkjahsdlgkjahlkdjahfsa'
url = 'https://www.website.com'
headers = {'api-key': api-key,
'Content-Type': 'application/json'}
request_data = {'foo': 'bar', 'egg': 'spam'}
result = requests.post(url, headers=headers, data=request_data)
The server is contacted. Suppose that instead I want to do something like
request_string = requests.foobar(url, headers=headers, data=request_data)
import os
os.system('curl ' + request_string)
So that I can look to see what the request is doing without bothering the server (possibly to the point that I could c&p it into curl), what would foobar be? Or in general, what is a way to inspect the contents of the request without making it?
Here's another post that implies that you can use Request().prepare() to observe the request without actually sending the request.
Furthermore the official documentation reads "In some cases you may wish to do some extra work to the body or headers (or anything else really) before sending a request. The simple recipe for this is the following" and then it illustrates Request.prepare()
Before downvoting/marking as duplicate, please note:
I have already tried out this, this, this, this,this, this - basically almost all the methods I could find pointed out by the Requests documentation but do not seem to find any solution.
Problem:
I want to make a POST request with a set of headers and form data.
There are no files to be uploaded. As per the request body in Postman, we set the parameters by selecting 'form-data' under the 'Body' section for the request.
Here is the code I have:
headers = {'authorization': token_string,
'content-type':'multipart/form-data; boundary=----WebKitFormBoundaryxxxxxXXXXX12345'} # I get 'unsupported application/x-www-form-url-encoded' error if I remove this line
body = {
'foo1':'bar1',
'foo2':'bar2',
#... and other form data, NO FILE UPLOADED
}
#I have also tried the below approach
payload = dict()
payload['foo1']='bar1'
payload['foo2']='bar2'
page = ''
page = requests.post(url, proxies=proxies, headers=headers,
json=body, files=json.dump(body)) # also tried data=body,data=payload,files={} when giving data values
Error
{"errorCode":404,"message":"Required String parameter 'foo1' is not
present"}
EDIT:
Adding a trace of the network console. I am defining it in the same way in the payload as mentioned on the request payload.
There isn't any gui at all? You could get the network data from chrome, although:
Try this:
headers = {'authorization': token_string}
Probably there is more authorization? Or smthng else?
You shouldn't add Content-Type as requests will handle it for you.
Important, you could see the content type as WebKitFormBoundary, so for the payload you must take, the data from the "name" variable.
Example:
(I know you won't upload any file, it just an example) -
So in this case, for my payload would look like this: payload = {'photo':'myphoto'} (yea there would be an open file etc etc, but I try to keep it simple)
So your payload would be this-> (So always use name from the WebKit)
payload = {'foo1':'foo1data',
'foo2':'foo2data'}
session.post(url,data = payload, proxies etc...)
Important! As I can see you use the method from requests library. Firstly you always should create a session like this
session = requests.session() -> it will handle cookies, headers, etc, and won't open a new session, or plain requests with every requests.get/post.
I am trying to send HTTP GET request using urllib/urllib2 with some data.
If we set some value of data param in urllib2.urlopen(url, data) the request object is changed to send POST request instead of GET.
Is there any way to achieve this? Standard or Hack?
Code snippet,
import requests
import urllib
query = urllib.urlencode({'query':'["=", ["fact", "role"], "storage"]'})
# using request object
print 'Output 1.'
response = requests.get("http://localhost:8082/v3/nodes", data=query)
print response.json()
print
# using urllib object
print 'Output 2.'
resp = urllib.urlopen('http://localhost:8082/v3/nodes', data=query)
print resp.read()
Output:
Output 1.
[{u'deactivated': None, u'facts_timestamp': u'2016-02-04T14:06:07.269Z', u'name': u'node_xx_11', u'report_timestamp': None, u'catalog_timestamp': u'2016-02-04T14:06:16.958Z'}, {u'deactivated': None, u'facts_timestamp': u'2016-02-04T14:06:05.865Z', u'name': u'node_xx_12', u'report_timestamp': None, u'catalog_timestamp': u'2016-02-04T14:06:13.614Z'}]
Output 2.
The POST method is not allowed for /v3/nodes
For the
References I have gone through,
https://docs.python.org/2/library/urllib2.html#urllib2.urlopen
https://docs.python.org/2/library/urllib2.html#urllib2.Request.add_data
This is not the road block for me, as I am able to use requests module for sending the data with GET request type. Curiosity is the reason of this post.
The data parameter of urlopen is used to set the body of the request. GET requests cannot contain a body, as they should only be used to return a resource, which should only be defined by it's URL.
If you need to pass parameters, you can append them to the url, in your case :
from urllib.request import urlopen
urlopen('http://localhost:8082/v3/nodes?{}'.format(query))
The data parameter is for POST only and you cannot send a body in a GET request, so if you want to specify parameters you have to pass them through the URL.
One easy way to build such an URL is through the help of urllib.urlencode. Take a look at the documentation for this function.
I created a chatbot that connects to a server and can read messages, now I'm at the point where I need to send messages, requiring request payload (according to the Network tab in Developer tools on google chrome). My opener consists of nothing but the following:
import urllib
import urllib2
from cookielib import CookieJar
self.cj = CookieJar()
self.opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(self.cj))
To stay and connected and read messages, I do the following, I do the following:
def connect(self,settings,xhr):
xhr_polling = self.get_code(xhr)
data = self.opener.open("http://chat2-1.wikia.com:80/socket.io/1/xhr-polling/" + xhr_polling + "?name=HairyBot&key=" +
settings['chatkey'] + "&roomId=" + str(settings['room']) + "&t=" + timestamp())
return data.read()
Settings consisting of the roomId and chatkey. The timestamp function creates a timestamp in accordance to what the servers needs (which isn't necessary to know for this question). Back to the question though, how can a payload be added to the opener to send a message to the chat?
As a suggestion, I recommend use the Requests library. It makes this stuff really simple:
import requests
session = requests.session() # For connection pooling
def connect(self,settings,xhr):
xhr_polling = self.get_code(xhr)
request = session.get('http://chat2-1.wikia.com:80/socket.io/1/xhr-polling/' + xhr_polling, params={
'name': 'HairyBot',
'key': settings['chatkey'],
'roomId': settings['room'],
't': timestamp()
})
return request.text
If you want to send a POST request instead, just change get to post and add some data:
def connect(self,settings,xhr):
xhr_polling = self.get_code(xhr)
request = session.post('http://chat2-1.wikia.com:80/socket.io/1/xhr-polling/' + xhr_polling, params={
'name': 'HairyBot',
'key': settings['chatkey'],
'roomId': settings['room'],
't': timestamp()
}, data={
'key': 'value'
})
return request.text
I'm not sure what you mean by "a payload", but presumably it's just another form variable named payload. If so, you send it the same way you do any other form variable, and you're already sending a bunch—roomId, t, etc.
One way of sending form variables is by URL-encoding them, tacking them onto the query string, and sending a GET request. That's what you're doing now. (It would be better to use proper urllib methods instead of hacking it together with string concatenation, but the end result is the same.)
The other way is by sending a POST body. The urllib2 documentation explains how to do this, and there are plenty of good examples online, but basically all you have to do is call urllib.urlencode() on your name-value pairs, then pass the result as the second argument (or as a keyword argument named data) to the open call.
In other words, something like this:
data = self.opener.open("http://chat2-1.wikia.com:80/socket.io/1/xhr-polling/" + xhr_polling,
urllib.urlencode(("name", "HairyBot"),
("key", settings['chatkey']),
("roomId", str(settings['room']),
("key", settings['chatkey']),
("t", timestamp()),
("payload", payload)))
Or, if you prefer, most servers will allow you to send some parameters on the query string and others in the POST data, so you can leave your existing code alone and just make one change:
data = self.opener.open("http://chat2-1.wikia.com:80/socket.io/1/xhr-polling/" + xhr_polling + "?name=HairyBot&key=" +
settings['chatkey'] + "&roomId=" + str(settings['room']) + "&t=" + timestamp(),
urllib.urlencode(("payload", payload)))