I'm trying to convert following curl request into pycurl:
curl -v
-H Accept:application/json \
-H Content-Type:application/json \
-d "{
name: 'abc',
path: 'def',
target: [ 'ghi' ]
}" \
-X POST http://some-url
I have following python code:
import pycurl, json
c = pycurl.Curl()
c.setopt(pycurl.URL, 'http://some-url')
c.setopt(pycurl.HTTPHEADER, ['Accept: application/json'])
data = json.dumps({"name": "abc", "path": "def", "target": "ghi"})
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.POSTFIELDS, data)
c.setopt(pycurl.VERBOSE, 1)
c.perform()
print curl_agent.getinfo(pycurl.RESPONSE_CODE)
c.close()
Executing this I had an error 415: Unsupported media type, so I have changed:
c.setopt(pycurl.HTTPHEADER, ['Accept: application/json'])
into:
c.setopt(pycurl.HTTPHEADER, [ 'Content-Type: application/json' , 'Accept: application/json'])
This time I have 400: Bad request. But bash code with curl works. Do you have any idea what should I fix in python code?
PycURL is a wrapper on the libcurl library written in C language so its Python API can be bit puzzling. As some people are advocating use of python requests instead I just want to point out that it isn't a perfect replacement. For me, its lack of DNS resolution timeout was a deal breaker. I also find it much slower on my Raspberry Pi. This comparison may be relevant:
Python Requests vs PyCurl Performance
So here's something that doesn't evade OP's question:
import pycurl
import json
from cStringIO import StringIO
curl = pycurl.Curl()
curl.setopt(pycurl.URL, 'http://some-url')
curl.setopt(pycurl.HTTPHEADER, ['Accept: application/json',
'Content-Type: application/json'])
curl.setopt(pycurl.POST, 1)
# If you want to set a total timeout, say, 3 seconds
curl.setopt(pycurl.TIMEOUT_MS, 3000)
## depending on whether you want to print details on stdout, uncomment either
# curl.setopt(pycurl.VERBOSE, 1) # to print entire request flow
## or
# curl.setopt(pycurl.WRITEFUNCTION, lambda x: None) # to keep stdout clean
# preparing body the way pycurl.READDATA wants it
# NOTE: you may reuse curl object setup at this point
# if sending POST repeatedly to the url. It will reuse
# the connection.
body_as_dict = {"name": "abc", "path": "def", "target": "ghi"}
body_as_json_string = json.dumps(body_as_dict) # dict to json
body_as_file_object = StringIO(body_as_json_string)
# prepare and send. See also: pycurl.READFUNCTION to pass function instead
curl.setopt(pycurl.READDATA, body_as_file_object)
curl.setopt(pycurl.POSTFIELDSIZE, len(body_as_json_string))
curl.perform()
# you may want to check HTTP response code, e.g.
status_code = curl.getinfo(pycurl.RESPONSE_CODE)
if status_code != 200:
print "Aww Snap :( Server returned HTTP status code {}".format(status_code)
# don't forget to release connection when finished
curl.close()
There are some more interesting features worth checking out in the libcurl curleasy setopts documentation
In your bash example, the property target is an array, in your Python example it is a string.
Try this:
data = json.dumps({"name": "abc", "path": "def", "target": ["ghi"]})
I also strongly advise you to check out the requests library which has a much nicer API:
import requests
data = {"name": "abc", "path": "def", "target": ["ghi"]}
response = requests.post('http://some-url', json=data)
print response.status_code
I know this is over a year old now, but please try removing the whitespace in your header value.
c.setopt(pycurl.HTTPHEADER, ['Accept:application/json'])
I also prefer using the requests module as well because the APIs/methods are clean and easy to use.
I had similar problem, and I used your code example but updated the httpheader section as follows:
c.setopt(pycurl.HTTPHEADER, ['Content-Type:application/json'])
It's better simple to use requests library. (http://docs.python-requests.org/en/latest)
I append python code for your original curl custom headers.
import json
import requests
url = 'http://some-url'
headers = {'Content-Type': "application/json; charset=xxxe", 'Accept': "application/json"}
data = {"name": "abc", "path": "def", "target": ["ghi"]}
res = requests.post(url, json=data, headers=headers)
print (res.status_code)
print (res.raise_for_status())
Related
I have go server that unmarshals the json it receives it.
It works when I do it using the curl but fails in case of python.
Go server Unmarshal code:
type Data struct {
Namespace string `json:"namespace"`
ContainerId string `json:"containerId"`
}
func notify(w http.ResponseWriter, r *http.Request) {
decoder := json.NewDecoder(r.Body)
var data Data
err := decoder.Decode(&data)
if err != nil {
glog.Errorf("Failed to decode the request json %s \n", err.Error())
return
}
...
}
If I Do curl command it works without complaining:
curl -i -H "Accept: application/json" -H "Content-Type:application/json" -X POST --data '{"namespace": "default", "containerId": "2f7c58d399f2dc35fa1be2abea19301c8e74973ddd72f55a778babf01db5ac26"}' http://mysvc:8080/notify
but if I do the same thing with Python it complains:
jsonPrep['containerId'] = "2f7c58d399f2dc35fa1be2abea19301c8e74973ddd72f55a778babf01db5ac26"
jsonPrep['namespace'] = "default"
headers = {'Content-type': 'application/json', 'Accept': 'application/json'}
r = requests.post('http://mysvc:8080/notify', json=json.dumps(jsonPrep), headers=headers)
the go server complains :
E1026 15:49:48.974117 1 main.go:59] Failed to decode the request json json: cannot unmarshal string into Go value of type main.Data
I don't see what is different when I do curl vs rest query in python.
Can anyone help me identify the issue?
The json argument to requests.post() is for passing a value that has not had json.dumps() called on it yet. requests calls json.dumps() on the json argument itself, so because you're passing json=json.dumps(jsonPrep), jsonPrep will end up being JSONified twice, which is not what you want.
Either use data:
requests.post(..., data=json.dumps(jsonPrep), ...)
or get rid of the json.dumps():
requests.post(..., json=jsonPrep, ...)
I'm trying to use Google's API for geolocation giving wifi data to determine location. This is their intro. And this is my code
#author: Keith
"""
import requests
payload = {
"considerIp": "false",
"wifiAccessPoints": [
{
"macAddress": "00:25:9c:cf:1c:ac",
"signalStrength": -43,
"signalToNoiseRatio": 0
},
{
"macAddress": "00:25:9c:cf:1c:ad",
"signalStrength": -55,
"signalToNoiseRatio": 0
}
],
'key':'<MyAPIKey>'
}
r = requests.post('https://www.googleapis.com/geolocation/v1/geolocate',
params=payload)
print(r.text)
This is the output
{
"location": {
"lat": 32.3643098,
"lng": -88.703656
},
"accuracy": 6061.0
}
The request ignored all of the payload except the key portion and just found the geolocation using my IP address. So I'm sending the json payload incorrectly. I know this is probably really simple, but I'm stuck and couldn't find an example of python being used with requests to do this type of API query. Thanks
Edit:
Picked up the cURL library and executed this command with success:
curl -d #your_filename.json -H "Content-Type: application/json" -i "https://www.googleapis.com/geolocation/v1/geolocate?key=<myapikey>"
and got the output I expected. I just want to be able to do the same thing in requests, but the data I'm trying to send is in "your_filename.json".
Please try the following:
r = requests.post('https://www.googleapis.com/geolocation/v1/geolocate?key='+ '<MyAPIKey>', json=payload)
Note the key was moved to query params (URL) and json argument was used in place of params.
Okay I figured out the error of my ways. Taking a better look at requests.post function, I see that I'm not using the parameters argument correctly. After this, it worked perfectly,
r = requests.post('https://www.googleapis.com/geolocation/v1/geolocate', params=parameters, json=mydata, headers=headers)
I am currently trying to integrate Stripe Connect and have come across the flowing CURl POST request:
curl https://connect.stripe.com/oauth/token \
-d client_secret=SECRET_CODE \
-d code="{AUTHORIZATION_CODE}" \
-d grant_type=authorization_code
However I am very new to CURL and have been doing some research and trying to use the requests package to do it. This is what my current code looks like:
data = '{"client_secret": "%s", "code": "%s", "grant_type": "authorization_code"}' % (SECRET_KEY, AUTHORIZATION_CODE)
response = requests.post('https://connect.stripe.com/oauth/token', json=data)
However this always returns a response code 400. I have no idea where I am going wrong and any guidance would be thoroughly appreciated.
The error is because you are passing your data as string, instead json param of requests.post call expects it to be dict. Your code should be:
import requests
data = {
"client_secret": SECRET_KEY,
"code": AUTHORIZATION_CODE,
"grant_type": "authorization_code"
}
response = requests.post('https://connect.stripe.com/oauth/token', json=data)
Take a look at request library's More complicated POST requests document.
I'm following this to restart Ambari components that are in INSTALLED state, for this i have written a python code parse the Ambari services using Pycurl that works.
But once the JSON is parsed i will be generating a JSON file like:
{
"RequestInfo": {
"command": "START",
"context": "Restart all components on HOST"
},
"Requests/resource_filters": [
{
"component_name": "NAMENODE",
"hosts": "hadoopm",
"service_name": "HDFS"
},
{
"component_name": "RESOURCEMANAGER",
"hosts": "hadoopm",
"service_name": "YARN"
}]}
this works with:
curl -u username:password -H 'X-Requested-By-ambari' http://ambariserver:8080/api/v1/clusters/CLUSTERNAME/requests -d#service-restart.json
but the same is not working and failing with 400 Bad request error with the below code:
import pycurl
c = pycurl.Curl()
c.setopt(pycurl.URL, url_post)
c.setopt(pycurl.HTTPHEADER, ["X-Requested-By:ambari"])
data = json.dumps(json.loads(open(output_temp_file,'rb').read()), indent=1, sort_keys=True)
tabs = re.sub('\n +', lambda match: '\n' + '\t' * (len(match.group().strip('\n')) / 2), data)
tabJSON=json.dumps(json.loads(open(output_temp_file,'rb').read()), indent=1, sort_keys=True)
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.USERPWD,'admin:'+admin_pass)
c.setopt(pycurl.POSTFIELDS, 'tabJSON')
c.setopt(pycurl.WRITEFUNCTION, service_buffer.write)
c.setopt(pycurl.VERBOSE, 1)
c.perform()
c.close()
and failing with HTTP/1.1 400 Bad Request
is there something wrong that i'm doing with this can someone please help me with this.
Probably API call format is outdated in documentation. I'd suggest going by example:
open dev console in Chrome
use Ambari UI to perform an action (e.g. restart all services)
go to Network, find relevant POST/PUT request (sort by non-200 Status column)
copy request (right click on request -> Copy -> copy as cURL )
Now you have up-to-date CURL command example, and can go further playing around request body
I was given a request in Bash and I have to translate it to Python 2.7. I did this kind of translations several times, but now I am not able to make it work and I do not understand why.
First of all, I was given this Bash request:
curl -X POST -v -u user#domain:password --data "#file.json" -H "Content-Type:application/json" http://destination_url_a
With the file file.json, whose content is the following one:
{
"username":"user#domain",
"password":"password",
"shortName":"a-short-name",
"visibility":"PRIVATE",
"sitePreset":"site-dashboard",
"title":"A Title",
"description":"A description."
}
If I execute the Bash line in my computer, the result is succesful.
As always, I tried to use requests library in Python to make it work. What I did is:
import requests
from requests.auth import HTTPBasicAuth
import json
data = {
"username": "user#domain",
"password": "password",
"shortName": "a-short-name",
"visibility": "PRIVATE",
"sitePreset": "site-dashboard",
"title": "A Title",
"description": "A description.",
}
headers = {'Content-Type': 'application/json'}
data_json = json.dumps(data)
r = requests.post(
url='http://destination_url_a',
data=data_json,
headers=headers,
auth=HTTPBasicAuth('user#domain', 'password'),
verify=False,
)
Unfortunately, the response, stored in r variable, is an error, despite the status code is 200.
What could be happening? Does anyone find a problem in my code or has any idea?
EDIT
However, this is another example very similar which worked perfectly:
Bash:
curl -v -H "Content-Type:application/json" -X POST --data "#file.json" -u user#domain:password http://destination_url_b
My Python code
import requests
from requests.auth import HTTPBasicAuth
import json
data = {
"userName": "user#domain",
"password": "password",
"firstName": "Firstname",
"lastName": "Lastname",
"email": "email#domain.com",
"disableAccount": "False",
"quota": -1,
"groups": ["a_group",],
}
headers = {'Content-Type': 'application/json'}
data_json = json.dumps(data)
r = requests.post(
url='http://destination_url_b',
data=data_json,
headers=headers,
auth=HTTPBasicAuth('user#domain', 'password'),
verify=False,
)
It seems to be almost the same to the other request, but this works. Different data is sent, and to a different subdomain (both are sent to the same domain). Will these modifications be important if we are talking about the User-Agent you mentioned?
Sometimes, webservices filter on user-agent. User agent of curl and python are different. That may explain.
Try to forge a "legitimate" user-agent by modifying the request header.
Finally, there was no error in the code neither in the User-Agent.
The problem was that the destination application did not accept sites with the same short-name value.
What I was doing is creating the site from Bash, which worked, then removing it from the interface of the app and trying to create it from Python with the same data. I was getting error when doing that because in spite of having removed the site, I had to remove some residual data of it from the trashcan of the app too. If not, app returned an error since it considered that the site I was trying to create already existed.
So if I had introduced different short-name in each attempt, I would not have had any error.