can anyone point out a tutorial that shows me how to do a POST request using urllib2 with the data being in JSON format?
Messa's answer only works if the server isn't bothering to check the content-type header. You'll need to specify a content-type header if you want it to really work. Here's Messa's answer modified to include a content-type header:
import json
import urllib2
data = json.dumps([1, 2, 3])
req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
f = urllib2.urlopen(req)
response = f.read()
f.close()
Whatever urllib is using to figure out Content-Length seems to get confused by json, so you have to calculate that yourself.
import json
import urllib2
data = json.dumps([1, 2, 3])
clen = len(data)
req = urllib2.Request(url, data, {'Content-Type': 'application/json', 'Content-Length': clen})
f = urllib2.urlopen(req)
response = f.read()
f.close()
Took me for ever to figure this out, so I hope it helps someone else.
Example - sending some data encoded as JSON as a POST data:
import json
import urllib2
data = json.dumps([1, 2, 3])
f = urllib2.urlopen(url, data)
response = f.read()
f.close()
To read json response use json.loads(). Here is the sample.
import json
import urllib
import urllib2
post_params = {
'foo' : bar
}
params = urllib.urlencode(post_params)
response = urllib2.urlopen(url, params)
json_response = json.loads(response.read())
You certainly want to hack the header to have a proper Ajax Request :
headers = {'X_REQUESTED_WITH' :'XMLHttpRequest',
'ACCEPT': 'application/json, text/javascript, */*; q=0.01',}
request = urllib2.Request(path, data, headers)
response = urllib2.urlopen(request).read()
And to json.loads the POST on the server-side.
Edit : By the way, you have to urllib.urlencode(mydata_dict) before sending them. If you don't, the POST won't be what the server expect
This is what worked for me:
import json
import requests
url = 'http://xxx.com'
payload = {'param': '1', 'data': '2', 'field': '4'}
headers = {'content-type': 'application/json'}
r = requests.post(url, data = json.dumps(payload), headers = headers)
Related
Im trying to mimic this POST request from this site with this payload:
from this URL: https://surviv.io/stats/gert1
Here is an image of the request im trying to mimic.
Here is my current code in python:
import requests
headers = {'content-type': 'application/json; charset=UTF-8'}
url = 'https://surviv.io/api/user_stats'
payload = {"slug":"gert1","interval":"all","mapIdFilter":"-1"}
r = requests.post(url=url, headers=headers, data=payload)
print(r.content)
This returns:
b'<html>\r\n<head><title>500 Internal Server Error</title></head>\r\n<body bgcolor="white">\r\n<center><h1>500 Internal Server Error</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'
This is not what I want it return. I want it to return the exact response shown in the response tab of the user_stats requests, which contains the player's stats.
This is what I want it to return:
{"slug":"gert1","username":"GERT","player_icon":"","banned":false,"wins":61,"kills":2830,"games":2034,"kpg":"1.4","modes":[{"teamMode":1,"games":1512,"wins":46,"kills":2230,"winPct":"3.0","mostKills":21,"mostDamage":1872,"kpg":"1.5","avgDamage":169,"avgTimeAlive":92},{"teamMode":2,"games":255,"wins":4,"kills":234,"winPct":"1.6","mostKills":8,"mostDamage":861,"kpg":"0.9","avgDamage":162,"avgTimeAlive":102},{"teamMode":4,"games":267,"wins":11,"kills":366,"winPct":"4.1","mostKills":17,"mostDamage":2225,"kpg":"1.4","avgDamage":246,"avgTimeAlive":125}]}
You should use the json attribute rather than data in the post method. r = requests.post(url=url, headers=headers, json=payload)
Change your code to following your forgot to use json :
import json
import requests
headers = {'content-type': 'application/json; charset=UTF-8'}
url = 'https://surviv.io/api/user_stats'
payload = {"slug":"gert1","interval":"all","mapIdFilter":"-1"}
r = requests.post(url=url, headers=headers, data=json.dumps(payload))
print(r.content)
I am trying to make a POST request to the following page: http://search.cpsa.ca/PhysicianSearch
In order to simulate clicking the 'Search' button without filling out any of the form, which adds data to the page. I got the POST header information by clicking on the button while looking at the network tab in Chrome Developer Tools. The reason I'm posting this instead of just copying solutions from the other similar problems is that I believe I may have not gotten the correct header information.
Is it properly formatted and did I grab the right information? I've never made a POST request before.
This is what I've managed to piece together:
import urllib.parse
import urllib.request
data = urllib.parse.urlencode({'Host': 'search.cpsa.ca', 'Connection': 'keep-alive', 'Content-Length': 23796,
'Origin': 'http://search.cpsa.ca', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'Cahce-Control': 'no-cache', 'X-Requested-With': 'XMLHttpRequest',
'X-MicrosoftAjax': 'Delta=true', 'Accept': '*/*',
'Referer': 'http://search.cpsa.ca/PhysicianSearch',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'en-GB,en-US;q=0.8,en;q=0.6',
'Cookie': 'ASP.NET_SessionId=kcwsgio3dchqjmyjtwue402c; _ga=GA1.2.412607756.1459536682; _gat=1'})
url = "http://www.musi-cal.com/cgi-bin/query?%s"
data = data.encode('ascii')
with urllib.request.urlopen("http://search.cpsa.ca/PhysicianSearch", data) as f:
print(f.read().decode('utf-8'))
This solution outputs the page's HTML, but not with any of the data I wanted to retrieve from the POST request.
This is how you do it.
from urllib import request, parse
data = parse.urlencode(<your data dict>).encode()
req = request.Request(<your url>, data=data) # this will make the method "POST"
resp = request.urlopen(req)
Thank you C Panda. You really made it easy for me to learn this module.
I released the dictionary that we pass does not encode for me. I had to do a minor change -
from urllib import request, parse
import json
# Data dict
data = { 'test1': 10, 'test2': 20 }
# Dict to Json
# Difference is { "test":10, "test2":20 }
data = json.dumps(data)
# Convert to String
data = str(data)
# Convert string to byte
data = data.encode('utf-8')
# Post Method is invoked if data != None
req = request.Request(<your url>, data=data)
# Response
resp = request.urlopen(req)
The above code encoded the JSON string with some extra \" that caused me a lot of problems. This looks like a better way of doing it:
from urllib import request, parse
url = "http://www.example.com/page"
data = {'test1': 10, 'test2': 20}
data = parse.urlencode(data).encode()
req = request.Request(url, data=data)
response = request.urlopen(req)
print (response.read())
Set method="POST" in request.Request().
Sending a POST request without a body:
from urllib import request
req = request.Request('https://postman-echo.com/post', method="POST")
r = request.urlopen(req)
content = r.read()
print(content)
Sending a POST request with json body:
from urllib import request
import json
req = request.Request('https://postman-echo.com/post', method="POST")
req.add_header('Content-Type', 'application/json')
data = {
"hello": "world"
}
data = json.dumps(data)
data = data.encode()
r = request.urlopen(req, data=data)
content = r.read()
print(content)
It failed when I use urlencode. So I use the following code to make a POST call in Python3:
from urllib import request, parse
data = b'{"parameter1": "test1", "parameter2": "test2"}'
req = request.Request("http://www.musi-cal.com/cgi-bin/query?%s", data)
resp = request.urlopen(req).read().decode('utf-8')
print(resp)
I'm working with wechat APIs ...
here I've to upload an image to wechat's server using this API
http://admin.wechat.com/wiki/index.php?title=Transferring_Multimedia_Files
url = 'http://file.api.wechat.com/cgi-bin/media/upload?access_token=%s&type=image'%access_token
files = {
'file': (filename, open(filepath, 'rb')),
'Content-Type': 'image/jpeg',
'Content-Length': l
}
r = requests.post(url, files=files)
I'm not able to post data
From wechat api doc:
curl -F media=#test.jpg "http://file.api.wechat.com/cgi-bin/media/upload?access_token=ACCESS_TOKEN&type=TYPE"
Translate the command above to python:
import requests
url = 'http://file.api.wechat.com/cgi-bin/media/upload?access_token=ACCESS_TOKEN&type=TYPE'
files = {'media': open('test.jpg', 'rb')}
requests.post(url, files=files)
Doc: https://docs.python-requests.org/en/master/user/quickstart/#post-a-multipart-encoded-file
In case if you were to pass the image as part of JSON along with other attributes, you can use the below snippet.
client.py
import base64
import json
import requests
api = 'http://localhost:8080/test'
image_file = 'sample_image.png'
with open(image_file, "rb") as f:
im_bytes = f.read()
im_b64 = base64.b64encode(im_bytes).decode("utf8")
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
payload = json.dumps({"image": im_b64, "other_key": "value"})
response = requests.post(api, data=payload, headers=headers)
try:
data = response.json()
print(data)
except requests.exceptions.RequestException:
print(response.text)
server.py
import io
import json
import base64
import logging
import numpy as np
from PIL import Image
from flask import Flask, request, jsonify, abort
app = Flask(__name__)
app.logger.setLevel(logging.DEBUG)
#app.route("/test", methods=['POST'])
def test_method():
# print(request.json)
if not request.json or 'image' not in request.json:
abort(400)
# get the base64 encoded string
im_b64 = request.json['image']
# convert it into bytes
img_bytes = base64.b64decode(im_b64.encode('utf-8'))
# convert bytes data to PIL Image object
img = Image.open(io.BytesIO(img_bytes))
# PIL image object to numpy array
img_arr = np.asarray(img)
print('img shape', img_arr.shape)
# process your img_arr here
# access other keys of json
# print(request.json['other_key'])
result_dict = {'output': 'output_key'}
return result_dict
def run_server_api():
app.run(host='0.0.0.0', port=8080)
if __name__ == "__main__":
run_server_api()
Use this snippet
import os
import requests
url = 'http://host:port/endpoint'
with open(path_img, 'rb') as img:
name_img= os.path.basename(path_img)
files= {'image': (name_img,img,'multipart/form-data',{'Expires': '0'}) }
with requests.Session() as s:
r = s.post(url,files=files)
print(r.status_code)
I confronted similar issue when I wanted to post image file to a rest API from Python (Not wechat API though). The solution for me was to use 'data' parameter to post the file in binary data instead of 'files'. Requests API reference
data = open('your_image.png','rb').read()
r = requests.post(your_url,data=data)
Hope this works for your case.
import requests
image_file_descriptor = open('test.jpg', 'rb')
# Requests makes it simple to upload Multipart-encoded files
files = {'media': image_file_descriptor}
url = '...'
requests.post(url, files=files)
image_file_descriptor.close()
Don't forget to close the descriptor, it prevents bugs: Is explicitly closing files important?
For Rest API to upload images from host to host:
import urllib2
import requests
api_host = 'https://host.url.com/upload/'
headers = {'Content-Type' : 'image/jpeg'}
image_url = 'http://image.url.com/sample.jpeg'
img_file = urllib2.urlopen(image_url)
response = requests.post(api_host, data=img_file.read(), headers=headers, verify=False)
You can use option verify set to False to omit SSL verification for HTTPS requests.
This works for me.
import requests
url = "https://example.com"
payload={}
files=[
('file',('myfile.jpg',open('/path/to/myfile.jpg','rb'),'image/jpeg'))
]
response = requests.request("POST", url, auth=("my_username","my_password"), data=payload, files=files)
print(response.text)
If you have CURL then you can directly get request from Postman.
import requests
url = "your URL"
payload={}
files=[
('upload_file',('20220212235319_1509.jpg',open('/20220212235319_1509.jpg','rb'),'image/jpeg'))
]
headers = {
'Accept-Language': 'en-US',
'Authorization': 'Bearer yourToken'
}
response = requests.request("POST", url, headers=headers, data=payload, files=files)
print(response.text)
With this code, urllib2 make a GET request:
#!/usr/bin/python
import urllib2
req = urllib2.Request('http://www.google.fr')
req.add_header('User-Agent', '')
response = urllib2.urlopen(req)
With this one (which is almost the same), a POST request:
#!/usr/bin/python
import urllib2
headers = { 'User-Agent' : '' }
req = urllib2.Request('http://www.google.fr', '', headers)
response = urllib2.urlopen(req)
My question is: how can i make a GET request with the second code style ?
The documentation (http://docs.python.org/release/2.6.5/library/urllib2.html) says that
headers should be a dictionary, and
will be treated as if add_header() was
called with each key and value as
arguments
Yeah, except that in order to use the headers parameter, you have to pass data, and when data is passed, the request become a POST.
Any help will be very appreciated.
Use:
req = urllib2.Request('http://www.google.fr', None, headers)
or:
req = urllib2.Request('http://www.google.fr', headers=headers)
Using Chris Atlee's python poster library is there any way to include cookie handling?
I have python http login code, which works with cookies:
cookiejar = cookielib.CookieJar()
urlOpener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
request = urllib2.Request(login_url, params)
result = urlOpener.open(request)
But when I need to upload a file, I don't know how to use both poster lib code and cookie handling code. Poster lib seems to need to call urllib2.urlopen() and not some custom url opener, like in the code above.
For instance, I don't know how to use cookies with the python file post code below:
register_openers()
params = {'file': open("test.txt", "rb"), 'name': 'upload test'}
datagen, headers = multipart_encode(params)
request = urllib2.Request(upload_url, datagen, headers)
result = urllib2.urlopen(request)
I sent an email to Chris AtLee asking whether we could get a basic authentication example. He was very cool about answering my questions and even ran some example code I sent him.
To include cookie handling, you do something like this:
opener = poster.streaminghttp.register_openers()
opener.add_handler(urllib2.HTTPCookieProcessor(cookielib.CookieJar())) # Add cookie handler
params = {'file': open("test.txt", "rb"), 'name': 'upload test'}
datagen, headers = poster.encode.multipart_encode(params)
request = urllib2.Request(upload_url, datagen, headers)
result = urllib2.urlopen(request)
To add basic authentication to the request, you simply do this (I added the base64 encode line for completeness):
opener = poster.streaminghttp.register_openers()
params = {'file': open("test.txt", "rb"), 'name': 'upload test'}
datagen, headers = poster.encode.multipart_encode(params)
request = urllib2.Request(upload_url, datagen, headers)
auth = base64.encodestring('%s:%s' % ('username', 'password'))[:-1] # This is just standard un/pw encoding
request.add_header('Authorization', 'Basic %s' % auth ) # Add Auth header to request
result = urllib2.urlopen(request)
Hope this helps. And another big thanks to Chris AtLee.
You don't have to modify the original source code, just install all required openers manually (without calling register_openers()):
import urllib2
import cookielib
import poster
handlers = [poster.streaminghttp.StreamingHTTPHandler(),
poster.streaminghttp.StreamingHTTPRedirectHandler(),
urllib2.HTTPCookieProcessor(cookielib.CookieJar())]
urllib2.install_opener(urllib2.build_opener(*handlers))
datagen, headers = poster.encode.multipart_encode({"image1": open("DSC0001.jpg", "rb")})
request = urllib2.Request("http://localhost:5000/upload_image", datagen, headers)
print urllib2.urlopen(request).read()
Have you tried this:
cookiejar = cookielib.CookieJar()
urlOpener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
register_openers()
params = {'file': open("test.txt", "rb"), 'name': 'upload test'}
datagen, headers = multipart_encode(params)
request = urllib2.Request(upload_url, datagen, headers)
result = urlOpener.open(request)
I've figured out how to do this. I'm not sure if this is the best way to go about things, but it works, so I'll share it. In order to use the poster lib with cookies one must add urllib2.HTTPCookieProcessor to the opener built in poster.streaminghttp.register_openers().
Essentially, modify poster.streaminghttp.register_openers() to look like the code below, and if you want to have cookie handling, pass in a cookiejar object.
def register_openers(cookiejar=None):
"""Register the streaming http handlers in the global urllib2 default
opener object.
Returns the created OpenerDirector object."""
handlers = [StreamingHTTPHandler, StreamingHTTPRedirectHandler]
if hasattr(httplib, "HTTPS"):
handlers.append(StreamingHTTPSHandler)
if cookiejar:
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar), *handlers)
else:
opener = urllib2.build_opener(*handlers)
urllib2.install_opener(opener)
return opener
Sample Usage:
# Logging in
import urllib, urllib2, cookielib
from poster.encode import multipart_encode
from poster.streaminghttp import register_openers
cookiejar = cookielib.CookieJar()
loginOpener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
params = urllib.urlencode({'username':'admin', 'password':'default'})
login_url = "http://127.0.0.1:8000/account/login/"
request = urllib2.Request(login_url, params)
login = loginOpener.open(request)
# Upload File
# use the login cookie for file upload
register_openers(cookiejar=cookiejar)
params = {'entity_file': open("test.txt", "rb"),'name': 'test', 'action':'upload'}
upload_url = "http://127.0.0.1:8000/upload/"
datagen, headers = multipart_encode(params)
request = urllib2.Request(upload_url, datagen, headers)
result = urllib2.urlopen(request)