I am trying to make a POST request to the following page: http://search.cpsa.ca/PhysicianSearch
In order to simulate clicking the 'Search' button without filling out any of the form, which adds data to the page. I got the POST header information by clicking on the button while looking at the network tab in Chrome Developer Tools. The reason I'm posting this instead of just copying solutions from the other similar problems is that I believe I may have not gotten the correct header information.
Is it properly formatted and did I grab the right information? I've never made a POST request before.
This is what I've managed to piece together:
import urllib.parse
import urllib.request
data = urllib.parse.urlencode({'Host': 'search.cpsa.ca', 'Connection': 'keep-alive', 'Content-Length': 23796,
'Origin': 'http://search.cpsa.ca', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'Cahce-Control': 'no-cache', 'X-Requested-With': 'XMLHttpRequest',
'X-MicrosoftAjax': 'Delta=true', 'Accept': '*/*',
'Referer': 'http://search.cpsa.ca/PhysicianSearch',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'en-GB,en-US;q=0.8,en;q=0.6',
'Cookie': 'ASP.NET_SessionId=kcwsgio3dchqjmyjtwue402c; _ga=GA1.2.412607756.1459536682; _gat=1'})
url = "http://www.musi-cal.com/cgi-bin/query?%s"
data = data.encode('ascii')
with urllib.request.urlopen("http://search.cpsa.ca/PhysicianSearch", data) as f:
print(f.read().decode('utf-8'))
This solution outputs the page's HTML, but not with any of the data I wanted to retrieve from the POST request.
This is how you do it.
from urllib import request, parse
data = parse.urlencode(<your data dict>).encode()
req = request.Request(<your url>, data=data) # this will make the method "POST"
resp = request.urlopen(req)
Thank you C Panda. You really made it easy for me to learn this module.
I released the dictionary that we pass does not encode for me. I had to do a minor change -
from urllib import request, parse
import json
# Data dict
data = { 'test1': 10, 'test2': 20 }
# Dict to Json
# Difference is { "test":10, "test2":20 }
data = json.dumps(data)
# Convert to String
data = str(data)
# Convert string to byte
data = data.encode('utf-8')
# Post Method is invoked if data != None
req = request.Request(<your url>, data=data)
# Response
resp = request.urlopen(req)
The above code encoded the JSON string with some extra \" that caused me a lot of problems. This looks like a better way of doing it:
from urllib import request, parse
url = "http://www.example.com/page"
data = {'test1': 10, 'test2': 20}
data = parse.urlencode(data).encode()
req = request.Request(url, data=data)
response = request.urlopen(req)
print (response.read())
Set method="POST" in request.Request().
Sending a POST request without a body:
from urllib import request
req = request.Request('https://postman-echo.com/post', method="POST")
r = request.urlopen(req)
content = r.read()
print(content)
Sending a POST request with json body:
from urllib import request
import json
req = request.Request('https://postman-echo.com/post', method="POST")
req.add_header('Content-Type', 'application/json')
data = {
"hello": "world"
}
data = json.dumps(data)
data = data.encode()
r = request.urlopen(req, data=data)
content = r.read()
print(content)
It failed when I use urlencode. So I use the following code to make a POST call in Python3:
from urllib import request, parse
data = b'{"parameter1": "test1", "parameter2": "test2"}'
req = request.Request("http://www.musi-cal.com/cgi-bin/query?%s", data)
resp = request.urlopen(req).read().decode('utf-8')
print(resp)
Related
Tried many ways to upload the data(postman, httpie etc as given on their site) on thingworx but not able to do that.
Please have a look on the following code to upload the data on thingworx:
import requests
import json
app_key = 'xxxx'
url = 'http://pp-1804040542ze.devportal.ptc.io/Thingworx/Things/lmtech_thing/Properties/humidity'
prms = {'appKey': app_key}
hdrs = {
'Accept': 'application/json',
'Content-Type': 'application/json'
}
data = {'humidiy': '20'}
text = json.dumps(data)
print 'data: ' + text
r = requests.put(url, params=prms, headers=hdrs, data=text)
print r.status_code
Have created thing and key successfully. but it always return 404 error.
Tried with postman too. Here are the screenshots as shown below:
The following code worked for me :-)
import requests # Import requests library to send requests to Thingworx
url = 'http://52.199.28.120:8080/Thingworx/Things/work_thing/Properties/temp'
# temp is one of my property name
value = 12 # Upload 12 on Thingworx
headers = {
'Content-Type': 'application/json',
'appkey': 'xxxxxxxxxxxxxxxxxxxxxx',
'Accept': 'application/json',
'x-thingworx-session': 'true',
'Cache-Control': 'no-cache',
}
data = {"temp": value} # JSON data to upload on Thingworx
response = requests.put(url, headers=headers, json=data)
# Note that we have to send put request
print 'Response Code:', response.status_code
# If 200 then data has been uploaded successfully
print 'Response Content:', response.content
Looking to connect to the Gooddata API and export a report via the API in python. The documentation is a bit confusing to follow.
I've defined a login to my instance of gooddata:
from urllib2 import Request, urlopen
import json
import requests
def login_gooddata(my_email, my_password):
url = 'https://secure.gooddata.com/gdc/account/login'
values = {
"postUserLogin": {
"login": my_email,
"password": my_password,
"remember": 0,
"verify_level": 0
}
}
headers = {
'Content-Type': 'application/json',
'Accept': 'application/json'
}
encoded_values = json.dumps(values)
#request = Request(url, data=encoded_values, headers=headers)
r = requests.post(url, data=encoded_values)
return r
That successfully logs me in, returning a 200 response.
Given the documentation from the gooddata website on connecting to the API, I'm trying to export a raw project file.
I set the project and object ids:
project_id = 'asibfakuyebkbhdbfaisdf'
object_id = '87234760'
values = {
"report_req": {
"reportDefinition": "/gdc/md/"+ project_id + "/obj/" + object_id
}
}
headers = {
'Accept': 'application/json',
'Content-Type': 'application/json'
}
url = 'https://secure.gooddata.com/gdc/app/projects/' + project_id + '/execute/raw/'
r = requests.post(url, data=json.dumps(values), headers=headers)
request = Request(url, data=json.dumps(values), headers=headers)
response_body = urlopen(requests).read()
print response_body
I played around with using r = requests.post(url, data=encoded_values and request = Request(url, data=encoded_values, headers=headers). Still receiving an error. I'm not really sure how to tackle next steps.
Following directions as stated in documentation for connecting to the API:
You need to perform all HTTP requests from a single "session" that remembers cookies from the login: perform s = requests.Session() once, then use s.post instead of requests.post.
See https://stackoverflow.com/a/31571805/3407728 for more.
I am completely new to Python.
I need to post an authentication request to a website using header and body details.
I have posted the code below that I have so far. When I run it, I get syntax error on this line:
req.add_header('API-KEY': 'my_api_key', 'ACCOUNT-ID': 'my_account_id', 'Content-Type': 'application/json; charset=UTF-8', 'VERSION': '2')
Can you please review and let me know where I've gone wrong?
import urllib.request
import json
body = {'identifier': 'my_id', 'password': 'my_encrypted_pwd', 'encryptedPassword': true}
url = 'https://mywebsite.com/gateway/deal/session'
req = urllib.request.Request(url)
req.add_header('API-KEY': 'my_api_key', 'ACCOUNT-ID': 'my_account_id', 'Content-Type': 'application/json; charset=UTF-8', 'VERSION': '2')
jsondata = json.dumps(body)
jsondataasbytes = jsondata.encode('UTF-8')
req.add_header('Content-Length', len(jsondataasbytes))
print (jsondataasbytes)
response = urllib.request.urlopen(req, jsondataasbytes)
I suggest you should use the requests module, as it's faster and simply better than urllib.request:
response = requests.put(
url,
body, headers={'API-KEY': 'my_api_key',
'ACCOUNT-ID': 'my_account_id',
'Content-Type': 'application/json; charset=UTF-8',
'VERSION': '2'
}
)
Now you can parse the response you get as usual.
I have an upload.php page that sends some data to a Python CGI script through a form, I then process the data in the background and I want to redirect to another php page, response_page.php, which displays info based on the processed data. I cannot send the data back to PHP and make the redirect at the same time, though.
My code:
#!/usr/bin/env python
import cgi
import cgitb
cgitb.enable()
try:
form = cgi.FieldStorage()
fn = form.getvalue('picture_name')
cat_id = form.getvalue('selected')
except KeyError:
print "Content-type: text/html"
print
print "<html><head>"
print "</head><body>error</body></html>"
else:
...
# here I processed the form data and stored it in data_to_be_displayed
# data to be processed and displayed in the response page
data_to_be_displayed = [1,2,3]
import httplib, json, urllib
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
conn = httplib.HTTPConnection('192.168.56.101:80')
#converting list to a json stream
data_to_be_displayed = json.dumps(data_to_be_displayed, ensure_ascii = 'False')
postData = urllib.urlencode({'matches':data_to_be_displayed})
conn.request("POST", "/response_page.php", postData, headers)
response = conn.getresponse()
if response.status == 200:
print "Location: /response_page.php"
print # to end the CGI response headers.
conn.close()
I found this: How to make python urllib2 follow redirect and keep post method , but I don't understand how I should use the urllib2.HTTPRedirectHandlerClass in my code.
Why don't you post to response_page.php using liburl2?
import urllib
import urllib2
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
data_to_be_displayed = json.dumps(data_to_be_displayed, ensure_ascii = 'False')
postData = urllib.urlencode({'matches':data_to_be_displayed})
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read()
For reference I've used the idea from pythons' documentation: https://docs.python.org/2/howto/urllib2.html#headers
you might also consider using Twisted apt for higher level code:
https://twistedmatrix.com/
EDIT:
After understanding better what are your asking for, I've found this post referring that redirect 307 is EXACTLY what you want (if now I understand right):
https://softwareengineering.stackexchange.com/questions/99894/why-doesnt-http-have-post-redirect
can anyone point out a tutorial that shows me how to do a POST request using urllib2 with the data being in JSON format?
Messa's answer only works if the server isn't bothering to check the content-type header. You'll need to specify a content-type header if you want it to really work. Here's Messa's answer modified to include a content-type header:
import json
import urllib2
data = json.dumps([1, 2, 3])
req = urllib2.Request(url, data, {'Content-Type': 'application/json'})
f = urllib2.urlopen(req)
response = f.read()
f.close()
Whatever urllib is using to figure out Content-Length seems to get confused by json, so you have to calculate that yourself.
import json
import urllib2
data = json.dumps([1, 2, 3])
clen = len(data)
req = urllib2.Request(url, data, {'Content-Type': 'application/json', 'Content-Length': clen})
f = urllib2.urlopen(req)
response = f.read()
f.close()
Took me for ever to figure this out, so I hope it helps someone else.
Example - sending some data encoded as JSON as a POST data:
import json
import urllib2
data = json.dumps([1, 2, 3])
f = urllib2.urlopen(url, data)
response = f.read()
f.close()
To read json response use json.loads(). Here is the sample.
import json
import urllib
import urllib2
post_params = {
'foo' : bar
}
params = urllib.urlencode(post_params)
response = urllib2.urlopen(url, params)
json_response = json.loads(response.read())
You certainly want to hack the header to have a proper Ajax Request :
headers = {'X_REQUESTED_WITH' :'XMLHttpRequest',
'ACCEPT': 'application/json, text/javascript, */*; q=0.01',}
request = urllib2.Request(path, data, headers)
response = urllib2.urlopen(request).read()
And to json.loads the POST on the server-side.
Edit : By the way, you have to urllib.urlencode(mydata_dict) before sending them. If you don't, the POST won't be what the server expect
This is what worked for me:
import json
import requests
url = 'http://xxx.com'
payload = {'param': '1', 'data': '2', 'field': '4'}
headers = {'content-type': 'application/json'}
r = requests.post(url, data = json.dumps(payload), headers = headers)