We Keep Coding

Using Python requests library for API GET request

I am trying to write a Python script that will allow me to accomplish what I normally would by using CURL to perform an API "GET". I have browsed through some questions on here but I am still a bit confused on how to do this; from what I have seen, I can use the "requests" library like this:
URL = requests.get("www.example.com/123")
However, the Curl command I normally run uses authentication as well. Here is an example:
curl -X GET -H 'Authorization: hibnn:11111:77788777YT666:CAL1' 'http://api.example.com/v1.11/user?id=123456
Can I still use the requests library when creating a script to pull this data? Specifically, I would need to be able to pass along the authentication info along with the URL itself.
Update:
Here is what my code looks like:
import requests
import json
url = ("api.example.com/v1.11/user?id=123456")
header = {"Authorization": "hibnn:11111:77788777YT666:CAL1"}
response = requests.get(url, headers=header)
print(response.json)
However, I am getting a response [401] error. I think I am probably doing something wrong with my headers, can anyone help?

You may pass headers as keyword argument to the get method.
>>> url = 'https://api.github.com/some/endpoint'
>>> headers = {'user-agent': 'my-app/0.0.1'}
>>> r = requests.get(url, headers=headers)
Reference

how to convert this curl command to some Python codes that do the same thing?

I am trying to download my data by using Fitbit API. I have figured out how to obtain a certain day's data, which is good. And here is the curl command I used:
curl -i -H "Authorization: Bearer (here goes a very long token)" https://api.fitbit.com/1/user/-/activities/heart/date/2016-6-14/1d/1sec/time/00:00/23:59.json >> heart_rate_20160614.json
However, I would like to collect hundreds of days' data and I don't want to do that manually. So I think I could use a Python loop. I read some other topics like this one and this one but still don't know how to 'translate' these curl commands into python language by using urllib2.
I have tried this:
import urllib2
url = 'https://api.fitbit.com/1/user/-/activities/heart/date/today/1d/1sec/time/00:00/00:01.json'
data = '{Authorization: Bearer (here goes a very long token)}'
req = urllib2.Request(url,data)
f = urllib2.urlopen(req)
but the got an error says "HTTP Error 404: Not Found"
So what is the correct way to 'translate' this curl command to python language? Thanks!

The problem comes from the construction of the Request object : by default, the second parameter is the data that you want to pass along with the request. Instead, you have to specify that you want to pass headers. This is the correct way to do it :
import urllib2
url = 'https://api.fitbit.com/1/user/-/activities/heart/date/2016-6-14/1d/1sec/time/00:00/23:59.json'
hdr = {'Authorization': 'Bearer (token)'}
req = urllib2.Request(url,headers=hdr)
f = urllib2.urlopen(req)
This wields a 401 on my side, but should work with your token.
You can have more informations on urllib2 (and the Request class) here
However, I suggest you take a look at Requests, which is in my opinion easier to use, and very well documented.
Hope it'll be helpful.

You can use the excellent lib requests which is much easier to use than urllib, in my opinion.
First, pip install requests, then in your interpreter:
import requests
response = requests.get(url='https://api.fitbit.com/1/user/-/activities/heart/date/2016-6-14/1d/1sec/time/00:00/23:59.json', headers={'Authorization':'Bearer <TOKEN>'})
if response.ok:
print response.content
else:
print "error", response.content
From here you can easily get the response content via response.content or response.json() if it's a JSON, and write it to a file.

SImple http post using python

I have a quick question to call an NLTK Api using python.To find the sentiment of "great";
The API syntax is
$ curl -d "text=great" http://text-processing.com/api/sentiment/
I need to use a python request to post this and to receive a json object as response.I am trying to use with
resp = requests.post(url, data=values, allow_redirects=True)
if url is http://text-processing.com/api/sentiment/
how must text parameter to be passed?

Modifying the guide from the requests documentation, to suit your requirement, this is what you do:
>>> import json
>>> url = 'http://text-processing.com/api/sentiment/'
>>> payload = {'text': 'great'}
>>> r = requests.post(url, data=json.dumps(payload))

Browser simulation - Python 3

I need to access a few HTML pages through a Python 3 script, problem is that I need COOKIE functionality, therefore a simple urllib HTTP request won't work.
Any ideas?

python3's urllib has cookie support, look at urllib.request.HTTPCookieProcessor, and http.cookiejar

Use requests.
>>> import requests
>>> url = 'http://httpbin.org/cookies/set/requests-is/awesome'
>>> r = requests.get(url)
>>> print r.cookies
{'requests-is': 'awesome'}
Reference: http://docs.python-requests.org/en/latest/user/quickstart/#cookies
As of a few days ago, requests supports Python 3, though you might have to use one of the develop branches, not entirely sure about the status of upstream integration.

How do I get JSON data from RESTful service using Python?

Is there any standard way of getting JSON data from RESTful service using Python?
I need to use kerberos for authentication.
some snippet would help.

I would give the requests library a try for this. Essentially just a much easier to use wrapper around the standard library modules (i.e. urllib2, httplib2, etc.) you would use for the same thing. For example, to fetch json data from a url that requires basic authentication would look like this:
import requests
response = requests.get('http://thedataishere.com',
auth=('user', 'password'))
data = response.json()
For kerberos authentication the requests project has the reqests-kerberos library which provides a kerberos authentication class that you can use with requests:
import requests
from requests_kerberos import HTTPKerberosAuth
response = requests.get('http://thedataishere.com',
auth=HTTPKerberosAuth())
data = response.json()

Something like this should work unless I'm missing the point:
import json
import urllib2
json.load(urllib2.urlopen("url"))

You basically need to make a HTTP request to the service, and then parse the body of the response. I like to use httplib2 for it:
import httplib2 as http
import json
try:
from urlparse import urlparse
except ImportError:
from urllib.parse import urlparse
headers = {
'Accept': 'application/json',
'Content-Type': 'application/json; charset=UTF-8'
}
uri = 'http://yourservice.com'
path = '/path/to/resource/'
target = urlparse(uri+path)
method = 'GET'
body = ''
h = http.Http()
# If you need authentication some example:
if auth:
h.add_credentials(auth.user, auth.password)
response, content = h.request(
target.geturl(),
method,
body,
headers)
# assume that content is a json reply
# parse content with the json module
data = json.loads(content)

If you desire to use Python 3, you can use the following:
import json
import urllib.request
req = urllib.request.Request('url')
with urllib.request.urlopen(req) as response:
result = json.loads(response.readall().decode('utf-8'))

Well first of all I think rolling out your own solution for this all you need is urllib2 or httplib2 . Anyways in case you do require a generic REST client check this out .
https://github.com/scastillo/siesta
However i think the feature set of the library will not work for most web services because they shall probably using oauth etc .. . Also I don't like the fact that it is written over httplib which is a pain as compared to httplib2 still should work for you if you don't have to handle a lot of redirections etc ..