I have been trying to use python to get data from an API - which I am able to access manually like below
curl --location --compressed --header “Authorization: Bearer [ACCESS_TOKEN]” [STREAM_URL]
How do I go about using it in python - I have read examples it says use requestsbut how to handle the Access_Token part in python
Any help would be appreciated?
Regards
VB
Just define your headers like you do in your curl example. In this case though, it's a dict. Assuming you want to use requests you would do something like this:
import requests
url = [STREAM_URL]
headers = {"Authorization": "Bearer [ACCESS TOKEN]"}
r = requests.get(url, headers=headers)
print(r.text)
There's no different what to use. urllib come with the python and if you don't like to download additional package, use urllib.
As OP mentioned --compressed in curl command, we should do the same to get gzipped content:
Edited ars answer:
import sys
if sys.version_info.major == 3:
import urllib.request as urllib2
else:
import urllib2
from StringIO import StringIO
import gzip
request = urllib2.Request('http://example.com/')
request.add_header('Accept-encoding', 'gzip')
request.add_header('Authorization', 'Bearer [ACCESS_TOKEN]')
response = urllib2.urlopen(request)
if response.info().get('Content-Encoding') == 'gzip':
buf = StringIO( response.read())
f = gzip.GzipFile(fileobj=buf)
data = f.read()
Does python urllib2 automatically uncompress gzip data fetched from webpage?
Related
Can someone please suggest the correct syntax for calling the below using python?
curl "https://sometest.api.token" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials&client_id={client_id}&client_secret={client_secret}"
My attempt:
import requests
import json
credentials='1111'
secret='2222'
url = 'https://sometest.api.token'
body = {'client_credentials':credentials,'client_secret':secret}
headers = {'Content-type': 'application/x-www-form-urlencoded'}
r = requests.post(url, data=json.dumps(body), headers=headers)
Due to documentation, if you want to send some form-encoded data, you simply pass a dictionary to the data argument.
So you have to try:
import requests
import json
credentials='1111'
secret='2222'
url = 'https://sometest.api.token'
body = {'client_credentials':credentials, 'client_secret':secret}
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
r = requests.post(url, data=body, headers=headers)
And also your parameters in python code are different from parameters in curl, maybe you have to check it.
I created a GET request in Python for an API and I would like to add headers and body
import urllib2
import os
proxy = 'http://26:Do#proxy:8080'
os.environ['http_proxy'] = proxy
os.environ['https_proxy'] = proxy
os.environ['HTTP_PROXY'] = proxy
os.environ['HTTPS_PROXY'] = proxy
contents = urllib2.urlopen("https://xxxx/lista?zile=50 ").read()
I tried in Postman and I received a response and I would like to receive the same response in python. How can I add headers and body ?
Thanks in advance
You can use the urlopen function with a Request object:
https://docs.python.org/2/library/urllib2.html#urllib2.urlopen
This Request object can contain headers and body:
https://docs.python.org/2/library/urllib2.html#urllib2.Request
Example: https://docs.python.org/2/howto/urllib2.html#data
P.S: HTTP GET requests don't have a body. Maybe you meant POST or PUT?
the best way is to use the request library which is pretty simple to use. https://realpython.com/python-requests/
example:
import requests
headers = {'Content-Type': 'application/json'}
data_json = {"some_key": "some_value"}
response = requests.post("https://xxxx/lista?zile=50", headers=headers, json=data_json)
I am attempting to upload a CSV file to an API (which gives me very little error info) using python requests library.
(I'm running Python 3.5 and using requests version 2.18.4 on OS X 10.11.6)
This curl command in terminal works fine: curl -F 'file=#/path/to/file.csv' myurl.com/upload -H "Authorization: TOKEN sometoken"
A multipart/form-data POST request from Postman also works, but I can't seem to make it work with the python requests library.
I've tried many variations of this request:
import requests
headers = {'Authorization': 'TOKEN sometoken', 'Content-Type': 'multipart/form-data'}
with open(file_path, 'rb') as f:
r = requests.post(myurl, headers=headers, data=f)
## I've also tried data={"file": f}
I get a status code of 200, but the response is {"success": "false"} (frustratingly unhelpful).
What am I missing in the python request compared to the curl request?
EDIT: it seems that the -F flag for the curl command emulates an HTML form that has been submitted...is there a way to do this with requests?
In your curl code you're using the -F parameter, which submits the data as a multipart message - in other words you're uploading a file.
With requests you can post files with the files parameter. An example:
import requests
headers = {'Authorization': 'TOKEN sometoken'}
data = {'file': open(file_path, 'rb')}
r = requests.post(myurl, headers=headers, files=data)
Note that requests creates the Content-Type header automatically, based on the submitted data.
The python open() function returns a file object. This is not what you want to send to the API.
Instead, you can use:
with open(file_path, 'rb') as f:
r = requests.post(myurl, headers=headers, data=f.read())
Syntax taken from here.
I'm having an issue converting a working cURL call to an internal API to a python requests call.
Here's the working cURL call:
curl -k -H 'Authorization:Token token=12345' 'https://server.domain.com/api?query=query'
I then attempted to convert that call into a working python requests script here:
#!/usr/bin/env python
import requests
url = 'https://server.domain.com/api?query=query'
headers = {'Authorization': 'Token token=12345'}
r = requests.get(url, headers=headers, verify=False)
print r
I get a HTTP 401 or 500 error depending on how I change the headers variable around. What I do not understand is how my python request is any different then the cURL request. They are both being run from the same server, as the same user.
Any help would be appreciated
Hard to say without knowing your api, but you may have a redirect that curl is honoring that requests is not (or at least isn't send the headers on redirect).
Try using a session object to ensure all requests (and redirects) have your header.
#!/usr/bin/env python
import requests
url = 'https://server.domain.com/api?query=query'
headers = {'Authorization': 'Token token=12345'}
#start a session
s = requests.Session()
#add headers to session
s.headers.update(headers)
#use session to perform a GET request.
r = s.get(url)
print r
I figured it out, it turns out I had to specify the "accept" header value, the working script looks like this:
#!/usr/bin/env python
import requests
url = 'https://server.domain.com/api?query=query'
headers = {'Accept': 'application/app.app.v2+json', 'Authorization': 'Token token=12345'}
r = requests.get(url, headers=headers, verify=False)
print r.json()
I want to use python urllib2 to simulate a login action, I use Fiddler to catch the packets and got that the login action is just an ajax request and the username and password is sent as json data, but I have no idea how to use urllib2 to send json data, help...
For Python 3.x
Note the following
In Python 3.x the urllib and urllib2 modules have been combined. The module is named urllib. So, remember that urllib in Python 2.x and urllib in Python 3.x are DIFFERENT modules.
The POST data for urllib.request.Request in Python 3 does NOT accept a string (str) -- you have to pass a bytes object (or an iterable of bytes)
Example
pass json data with POST in Python 3.x
import urllib.request
import json
json_dict = { 'name': 'some name', 'value': 'some value' }
# convert json_dict to JSON
json_data = json.dumps(json_dict)
# convert str to bytes (ensure encoding is OK)
post_data = json_data.encode('utf-8')
# we should also say the JSON content type header
headers = {}
headers['Content-Type'] = 'application/json'
# now do the request for a url
req = urllib.request.Request(url, post_data, headers)
# send the request
res = urllib.request.urlopen(req)
# res is a file-like object
# ...
Finally note that you can ONLY send a POST request if you have SOME data to send.
If you want to do an HTTP POST without sending any data, you should send an empty dict as data.
data_dict = {}
post_data = json.dumps(data_dict).encode()
req = urllib.request.Request(url, post_data)
res = urllib.request.urlopen(req)
import urllib2
import json
# Whatever structure you need to send goes here:
jdata = json.dumps({"username":"...", "password":"..."})
urllib2.urlopen("http://www.example.com/", jdata)
This assumes you're using HTTP POST to send a simple json object with username and password.
You can specify data upon request:
import urllib
import urllib2
url = 'http://example.com/login'
values = YOUR_CREDENTIALS_JSON
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
You can use the 'requests' python library to achieve this:
http://docs.python-requests.org/en/latest/index.html
You will find this example:
http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests (More complicated POST requests)
>>> import requests
>>> payload = {'key1': 'value1', 'key2': 'value2'}
>>> r = requests.post("http://httpbin.org/post", data=payload)
It seems python do not set good headers when you are trying to send JSON instead of urlencoded data.