I would like to convert this curl command to something that I can use in Python for an existing script.
curl -u 7898678:X -H 'Content-Type: application/json' \
-d '{"message":{"body":"TEXT"}}' http://sample.com/36576/speak.json
TEXT is what i would like to replace with a message generated by the rest of the script.(Which is already working reasonable, although I don't think it is following best practices or particularity reliable. - need to find out how to properly learn to program (ie not use google for assembling stuff))
I would like this to work with the standard library if possible.
I would like this to work with the standard library if possible.
The standard library provides urllib and httplib for working with URLs:
>>> import httplib, urllib
>>> params = urllib.urlencode({'apple': 1, 'banana': 2, 'coconut': 'yummy'})
>>> headers = {"Content-type": "application/x-www-form-urlencoded",
... "Accept": "text/plain"}
>>> conn = httplib.HTTPConnection("example.com:80")
>>> conn.request("POST", "/some/path/to/site", params, headers)
>>> response = conn.getresponse()
>>> print response.status, response.reason
200 OK
If you want to execute curl itself, though, you can just invoke os.system():
import os
TEXT = ...
cmd = """curl -u 7898678:X -H 'Content-Type: application/json'""" \
"""-d '{"message":{"body":"%{t}"}}' http://sample.com/36576/speak.json""" % \
{'t': TEXT}
If you're willing to relax the standard-library-only restriction, you can use PycURL. Beware that it isn't very Pythonic (it's pretty much just a thin veneer over libcurl), and I'm not sure how compatible it is with Python 3.
While there are ways to handle authentication in urllib2, if you're doing Basic Authorization (which means effectively sending the username and password in clear text) then you can do all of what you want with a urllib2.Request and urllib2.urlopen:
import urllib2
def basic_authorization(user, password):
s = user + ":" + password
return "Basic " + s.encode("base64").rstrip()
req = urllib2.Request("http://localhost:8000/36576/speak.json",
headers = {
"Authorization": basic_authorization("7898678", "X"),
"Content-Type": "application/json",
# Some extra headers for fun
"Accept": "*/*", # curl does this
"User-Agent": "my-python-app/1", # otherwise it uses "Python-urllib/..."
},
data = '{"message":{"body":"TEXT"}}')
f = urllib2.urlopen(req)
I tested this with netcat so I could see that the data sent was, excepting sort order, identical in both cases. Here the first one was done with curl and the second with urllib2
% nc -l 8000
POST /36576/speak.json HTTP/1.1
Authorization: Basic Nzg5ODY3ODpY
User-Agent: curl/7.19.4 (universal-apple-darwin10.0) libcurl/7.19.4 OpenSSL/0.9.8k zlib/1.2.3
Host: localhost:8000
Accept: */*
Content-Type: application/json
Content-Length: 27
{"message":{"body":"TEXT"}} ^C
% nc -l 8000
POST /36576/speak.json HTTP/1.1
Accept-Encoding: identity
Content-Length: 27
Connection: close
Accept: */*
User-Agent: my-python-app/1
Host: localhost:8000
Content-Type: application/json
Authorization: Nzg5ODY3ODpY
{"message":{"body":"TEXT"}}^C
(This is slightly tweaked from the output. My test case didn't use the same url path you used.)
There's no need to use the underlying httplib, which doesn't support things that urllib2 gives you like proxy support. On the other hand, I do find urllib2 to be complicated outside of this simple sort of request and if you want better support for which headers are sent and the order they are sent then use httplib.
Thanks every
this works
import urllib2
def speak(status):
def basic_authorization(user, password):
s = user + ":" + password
return "Basic " + s.encode("base64").rstrip()
req = urllib2.Request("http://example.com/60/speak.json",
headers = {
"Authorization": basic_authorization("2345670", "X"),
"Content-Type": "application/json",
"Accept": "*/*",
"User-Agent": "my-python-app/1",
},
data = '{"message":{"body":'+ status +'}}')
f = urllib2.urlopen(req)
speak('Yay')
Take a look at pycurl http://pycurl.sourceforge.net/
Related
I'm trying to turn this cURL request into a Python code. I want to eventually be able to save this to a CSV file but I need to get connected first.
curl --compressed -H 'Accept: application/json' -H 'X-Api-Key: 123abc' 'https://us.market-api.kaiko.io/v2/data/trades.v1/exchanges/cbse/spot/btc-usd/aggregations/count_ohlcv_vwap?interval=1h'
I started with this:
import requests
import json
key='api-key'
url = 'https://us.market-api.kaiko.io/v2/data/trades.v1/exchanges/'
s = requests.Session()
s.auth = (key)
headers = {
*not sure how to do this*
}
r = requests.get(url, headers=headers)
The docs say this needs to be in the header:
Accept: application/json
Accept-Encoding: gzip:
How do I include the API key? how do I save the data once its returned?
X-Api-Key would be a request header, so you can include it in your headers variable, like this:
headers = {
"X-Api-Key": key,
"Accept": "application/json",
"Accept-Encoding": "gzip"
}
(took the others ones from your current curl request)
You can get the data by using r.text, like this:
print(r.text)
Your code should look like this:
import requests
import json
key='api-key'
url = 'https://us.market-api.kaiko.io/v2/data/trades.v1/exchanges/'
headers = {
"X-Api-Key": key,
"Accept": "application/json",
"Accept-Encoding": "gzip"
}
r = requests.get(url, headers=headers)
print(r.text)
If you want to get a json object instead, you can use r.json()
I am getting http code 400 (bad request) when I am trying to post a file to server using python requests library.
Corresponding curl request which is successful:
curl -X POST -i https://de.api.labs.sophos.com/analysis/file/static/v1 \
-H 'Authorization: auth_string' \
-H 'Content-Type: multipart/form-data' \
-F "file=#filename"
API documentation: https://api.labs.sophos.com/doc/analysis/file/static.html
Can someone help me out what I might be doing wrong?
My code so far:
import requests
url = "https://de.api.labs.sophos.com/analysis/file/static/v1"
headers = {'content-type': 'multipart/form-data', 'Authorization': authorization}
with open(filepath, 'rb') as f:
files = {'file': f} # Even tried {'file': f.read()}
r = requests.post(url, files=files, headers=headers)
if r.status_code in [200, 202]:
return r.json()
else:
return r
TL;DR
Try to do it this way:
import requests
url = "https://de.api.labs.sophos.com/analysis/file/static/v1"
headers = {'Authorization': authorization} # no Content-Type here
r = requests.post(url, headers=headers, files={"file": open(filepath, "rb")})
print(r.status_code, r.text)
Why
You shouldn't set Content-Type header manually when posting files with requests.
There are 2 reasons why:
requests will set Content-Type to multipart/form-data implicitly before making an actual HTTP request (as it does for Content-Length for example)
When using Content-Type: multipart/form-data you should specify a boundary as well. If no boundary is set, server will not be able to read data from request body correctly. So boundary is a required part of Content-Type header, if you use multipart/form-data.
In your example you haven't set boundary for the request. The fact is that requests does not set it for you if you override Content-Type header (which you do). And then server is not able to read your file in request body. Therefore, it returns you 400 Bad Request.
You can check it by typing print(r.request.headers["Content-Type"]) after you've made your request. It will output this:
multipart/form-data
, but it must look like this instead:
multipart/form-data; boundary=6387a52fb4d1465310a2b63b2d1c6e70
On the other hand, curl adds boundary implicitly, so you everything is fine and you receive 200 OK.
You can check it as well:
curl -H 'Content-Type: multipart/form-data' -F "file=#123.txt" -v http://httpbin.org/post
Which outputs:
* Connected to httpbin.org (34.230.136.58) port 80 (#0)
> POST /post HTTP/1.1
> Host: httpbin.org
> User-Agent: curl/7.47.0
> Content-Type: multipart/form-data; boundary=------------------------d257f5f4377a3997
...
I'm trying to create a Python-Requests version of the following curl POST command (which works perfectly and provides the expected response):
curl -F 'json={"method":"update_video","params":{"video":{"id":"129263001","itemState":"INACTIVE"},"token":"jCoXH5OAMYQtXm1sg62KAF3ysG90YLagDAdlhg.."}}' https://api.somewebservice.com/services/post
Using:
curl -v -F 'json={"method":"update_video","params":{"video":{"id":"582984001","itemState":"INACTIVE"},"token":"jCoXH5OAMYQtXm1sg62KAF3ysG90YLagEECDAdlhg.."}}' https://api.somewebservice.com/services/post
I get the following (only including output after all the TLS handshakes, server certificate data, etc):
....
> POST /services/post HTTP/1.1
> User-Agent: curl/7.41.0
> Host: api.somewebservice.com
> Accept: */*
> Content-Length: 294
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=------------------------871a9aa84d3c0de2
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Content-Type: application/json;charset=UTF-8
< Content-Length: 1228
< Date: Sun, 10 Apr 2016 07:04:00 GMT
< Server: somewebservice
Given that the above cURL command works perfectly and given this output here running in verbose mode, am I correct in assuming that what I need to do is take a multi-part/form approach that sends a JSON object in a form if I'm trying to recreate this using Python-Requests?
So far, I've tried:
import requests
import json
def deactivate_request():
url = "https://api.somewebservice.com/services/post"
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
payload = {"method":"update_video","params":{"video":{"id":"12926301","itemState":"INACTIVE"},"token":"jCoXH5OKAF3ysG90YLagEECTP16uOUSg_fEGDAdlhg.."}}
r = requests.post(url, json=payload, headers=headers)
print(r.text)
I've tried different variations too, like:
r = requests.post(url, data=json.dumps(payload), headers=headers)
or without headers, like this:
r = requests.post(url, data=json.dumps(payload))
or this:
r = requests.post(url, json=payload)
And nothing seems to work, I just keep getting the same error message:
{"error": {"name":"MissingJSONError","message":"Could not find JSON-RPC.","code":211}, "result": null, "id": null}
The documentation for this web service for that "211" error states that:
We got a null string for either the json parameter (for a non-multipart post) or the first part of a multipart post.
What am I doing wrong here in terms of recreating this cURL request using the Requests module? I thought that I could send the payload object as form-encoded data, and it looks like that is what the cURL command is doing with the -F argument there.
Apparently this curl command can be recreated with the following:
import requests
def deactivate_request():
url = "https://api.somewebservice.com/services/post"
print url
#headers = {"Authorization": "Bearer " + token, "Content-Type": "application/json"}
headers = {'Content-Type': 'application/json'}
print(headers)
payload = 'json={"method":"update_video", "params":{"video":{"id":"620001", "itemState":"INACTIVE"}, "token":"jCoXH5OAMYQtXm1sg62KAF3yECTP16uOUSg_fEGDAdlhg.."}}'
# using params instead of data because we are making this POST request by
# constructing query string URL with key/value pairs in it.
r = requests.post(url, params=payload, headers=headers)
Not quite obvious as the curl command uses 'multipart/form-data' in its header whereas with the above we're just using 'application/json'.
I'm trying to POST some data to an HTTPS server. It requires a very particular set of headers. I'm able to complete the request, but I'm unable to do so in Python.
The curl:
curl -i
-H "Authorization: Basic a2V5OnNlY3JldA=="
-H "Content-Type: application/x-www-form-urlencoded"
-H "Content-Length: 99"
-H "User-Agent: Dalvik/2.1.0 (Linux; U; Android 5.1.1; Nexus 6 Build/LMY48Y)"
-H "Host: test.example.com"
-H "Connection: Keep-Alive"
-H "Content-type: application/json"
-d "grant_type=password&username=me%40example.com&password=abcd*1234&scope=scope1_services+scope1_data"
"https://test.example.com/login/get/token/"
The Python is
import httplib, urllib
host = "test.example.com"
url = "/login/get/token/"
params = urllib.urlencode({"grant_type":"password", "username":"me#example.com", "password":"abcd*1234", "scope":"scope1_services+scope1_data" })
headers = {"Authorization": "Basic a2V5OnNlY3JldA==", "Content-type": "application/x-www-form-urlencoded", "Content-Length":"99", "User-Agent":"Dalvik/2.1.0 (Linux; U; Android 5.1.1; Nexus 6 Build/LMY48Y)", "Host":host, "Connection": "Keep-Alive", "Content-type":"application/json"}
conn = httplib.HTTPSConnection(host)
conn.request("POST", url, params, headers)
response = conn.getresponse()
print response.status, response.reason
I just end up with 400 Bad Request and the error message
{
"error":"unsupported_grant_type",
"error_description":"The authorization grant type is not supported by the authorization server."
}
As far as I can tell, everything should be the same.
I've tried manually encoding the POST payload as params="grant_type=password&user... but I still get the same error.
Any idea what incredibly obvious thing I'm missing?
I think it is actually Content-Type not Content-type. Watch out the capital T there.
Also, I think you can remove these safely from the header:
"Content-Length":"99",
"Host":host,
"Connection": "Keep-Alive",
And, are you sure you need this? You are not posting any json here!
"Content-type":"application/json"
I`m trying to make those 2 requests in python:
Request 1:
curl -X POST -H "Content-Type: application/json" -d '{ "auth_token": "auth1", "widget": "id1", "title": "Something1", "text": "Some text", "moreinfo": "Subtitle" }' serverip
Request 2:
vsphere_dict = {}
vsphere_dict['server_name'] = "servername"
vsphere_dict['api_version'] = apiVersion
vsphere_dict['guest_count'] = guestCount
vsphere_dict['guest_on'] = guestOnLen
vsphere_dict['guest_off'] = guestOffLen
#Convert output to Json to be sent
data = json.dumps(vsphere_dict)
curl -X POST -H "Content-Type: application/json" -d 'data' serverip
Neither of them seems to work. Is there any way I can send them in Python?
Update:
The part that I cannot handle is the pass auth and widget. I have tried the following without success:
import urllib2
import urllib
vsphere_dict = dict(
server_name="servername",
api_version="apiVersion",
guest_count="guestCount",
guest_on="guestOnLen",
guest_off="guestOffLen",
)
url = "http://ip:port"
auth = "authid89"
widget = "widgetid1"
# create request object, set url and post data
req = urllib2.Request(auth,url, data=urllib.urlencode(vsphere_dict))
# set header
req.add_header('Content-Type', 'application/json')
# send request
response = urllib2.urlopen(req)**
Resulting in "urllib2.HTTPError: HTTP Error 500: Internal Server Error"
Any ideas how I can pass the auth and widget correctly?
UPDATE:
To see what is different I have started a nc server locally. Here are the results:
Correct curl request using this code:
curl -X POST -H "Content-Type: application/json" -d '{ "auth_token": "auth", "widget": "widgetid", "title": "Something", "text": "Some text", "moreinfo": "Subtitle" }' http://localhost:8123
sends this which does work:
POST / HTTP/1.1
User-Agent: curl/7.21.0 (i386-redhat-linux-gnu) libcurl/7.21.0 NSS/3.12.10.0 zlib/1.2.5 libidn/1.18 libssh2/1.2.4
Host: localhst:8123
Accept: */*
Content-Type: application/json
Content-Length: 165
{ "auth_token": "token", "widget": "widgetid", "title": "Something", "text": "Some text", "moreinfo": "Subtitle" }
And request using this code
import requests
import simplejson as json
url = "http://localhost:8123"
data = {'auth_token': 'auth1', 'widget': 'id1', 'title': 'Something1', 'text': 'Some text', 'moreinfo': 'Subtitle'}
headers = {'Content-type': 'application/json'}
r = requests.post(url, data=json.dumps(data), headers=headers)
sends this which does not work:
POST / HTTP/1.1
Host: localhst:8123
Content-Length: 108
Content-type: application/json
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/2.0.1 CPython/2.7.0 Linux/2.6.35.14-106.fc14.i686
{"text": "Some text", "auth_token": "auth1", "moreinfo": "Subtitle", "widget": "id1", "title": "Something1"}
Requests provides you with the simplest and yet (very) powerful way to deal with HTTP requests in Python.
Maybe try something like this:
import requests
import simplejson as json
url = "http://ip:port"
data = {'auth_token': 'auth1', 'widget': 'id1', 'title': 'Something1', 'text': 'Some text', 'moreinfo': 'Subtitle'}
headers = {'Content-type': 'application/json'}
r = requests.post(url, data=json.dumps(data), headers=headers)
If the API requests authentication:
r = requests.post(url, data=json.dumps(data), headers=headers, auth=('user', 'pass'))
See [Requests auth] for details.
Well sure, using Python-Requests which is a Python library for sending requests like Curl. You can take a look at the Complicated Post Requests section.
Or, if you'd like to use curl inside of Python, you can use pyCurl.
In the example from the Dashing website, they use:
curl -d '{ "auth_token": "YOUR_AUTH_TOKEN", "current": 100 }' http://localhost:3030/widgets/karma
From the cURL man page, maybe you need to post it as form-urlencoded?
-d, --data
(HTTP) Sends the specified data in a POST request to the HTTP server, in the same way that a browser does when a user has filled in an HTML form and presses the submit button. This will cause curl to pass the data to the server using the content-type application/x-www-form-urlencoded. Compare to -F, --form.
-d, --data is the same as --data-ascii. To post data purely binary, you should instead use the --data-binary option. To URL-encode the value of a form field you may use --data-urlencode.
If any of these options is used more than once on the same command line, the data pieces specified will be merged together with a separating &-symbol. Thus, using '-d name=daniel -d skill=lousy' would generate a post chunk that looks like 'name=daniel&skill=lousy'.
If you start the data with the letter #, the rest should be a file name to read the data from, or - if you want curl to read the data from stdin. Multiple files can also be specified. Posting data from a file named 'foobar' would thus be done with --data #foobar. When --data is told to read from a file like that, carriage returns and newlines will be stripped out.
You might also want to try python-requests http://requests.readthedocs.org/en/latest/user/quickstart/#more-complicated-post-requests
Update: I got it to work
import requests
import json
payload = {'auth_token': 'YOUR_AUTH_TOKEN', 'title': "pythontest"}
r = requests.post("http://localhost:3030/widgets/welcome", data=json.dumps(payload))
print r.text
You need to post the json like a form.
why not use urllib2?
import urllib2
import urllib
vsphere_dict = dict(
server_name="servername",
api_version=apiVersion,
guest_count=guestCount,
guest_on=guestOnLen,
guest_off=guestOffLen,
)
# create request object, set url and post data
req = urllib2.Request(some_url, data=urllib.urlencode(vsphere_dict))
# set header
req.add_header('Content-Type', 'application/json')
# send request
response = urllib2.urlopen(req)
UPD:
sorry, by i not understand that is auth and widget. Maybe this is also POST data?
HTTP Error 500 - can mean that server received not all POST parameters.