I have been trying to learn Solr4.0, and I am looking at the JSON document update in their documentation which looks like the following:
cd example/exampledocs
curl 'http://localhost:8983/solr/update/json?commit=true' --data-binary #books.json -H 'Content-type:application/json'
It works fine and I can see the updated documents on my solr index. However, I was wondering how I can use this curl command in python via urllib2. So, something like:
theurl=r"""http://localhost:8983/solr/update/json?commit=true --data-binary #books.json -H 'Content-type:application/json'"""
import urllib2
import httplib
import cookielib
...use urllib2 to post theurl
However, this does not work. It looks like urllib2 does not recognize (such as the -H which obviously seems curl specific) the above formed theurl. How should one be formatting theurl such that I could use it with urllib2?
I would try
import urllib2
with open('books.json', 'rb') as data_file:
my_data = data_file.read()
req = urllib2.Request(url='http://localhost:8983/solr/update/json?commit=true',
data=my_data)
req.add_header('Content-type', 'application/json')
f = urllib2.urlopen(req)
# Begin using data like the following
print f.read()
From this you can see that the --data-binary parameter is just data sent to the server like in a POST request. When that parameter begins with the # sign, it means to read data from a file. In this case, it's the file 'books.json'. You also need to send the header (the -H parameter of curl). So you only need to call the add_header method with the header name and its value.
Hope that gets you started. More info about urllib2 can be found at http://docs.python.org/2/library/urllib2.html
Because urllib2 is not available in Python 3.x I offer this alternative.
This code snippet worked for me using Python 3.3 and the excellent requests library
import requests
def postXml(host, xmlFile):
url = "http://%s:8983/solr/update" % host
headers = {"content-type" : "text/xml" }
params = {"commit" : "false" }
payload = open(xmlFile, "rb").read()
r = requests.post(url, data=payload, params=params, headers=headers)
print("got back: %s" % r.text)
Related
I am trying to recreate a Curl command from the Gentle forced aligner:
https://github.com/lowerquality/gentle
Curl command:
curl -F "audio=#audio.mp3" -F "transcript=#words.txt" "http://localhost:8765/transcriptions?async=false"
Code so far:
import requests
url = 'http://localhost:8765/transcriptions/'
files = {'audio': open('C://Users//user//Desktop//gentle//audio.mp3','rb'), 'transcript':
open('C://Users//andrey_user//Desktop//gentle//words.txt', 'rb')}
headers = {}
response = requests.post(url,
files=files,
headers=headers)
print(response.text)
But it only returns the HTML and says it is a GET request, however the curl command gives me the correct result. Thanks for your help!
Try this code :
import requests
params = (
('async', 'false'),
)
files = {
'audio': ('audio.mp3', open('audio.mp3', 'rb')),
'transcript': ('words.txt', open('words.txt', 'rb')),
}
response = requests.post('http://localhost:8765/transcriptions', params=params, files=files)
print(response.text)
I believe you also have to convert the response to JSON, assuming that the library you're trying to send the request to is a REST API. I'm not sure why it says its a get request, but if that's the case, just try changing it to a get request and see if it works out.
So adding r1=response.json() should give you access to the correct JSON data
I need to send python requests data in application/x-www-form-urlencoded. Couldn;t find the answer. It must be that format otherwise the web won;t pass me :(
simple request should work
import requests
url = 'application/x-www-form-urlencoded&username=login&password=password'
r = requests.get(url)
or a JSON post:
import requests
r = requests.post('application/x-www-form-urlencoded', json={"username": "login","password": password})
I am trying to write a Python script that will allow me to accomplish what I normally would by using CURL to perform an API "GET". I have browsed through some questions on here but I am still a bit confused on how to do this; from what I have seen, I can use the "requests" library like this:
URL = requests.get("www.example.com/123")
However, the Curl command I normally run uses authentication as well. Here is an example:
curl -X GET -H 'Authorization: hibnn:11111:77788777YT666:CAL1' 'http://api.example.com/v1.11/user?id=123456
Can I still use the requests library when creating a script to pull this data? Specifically, I would need to be able to pass along the authentication info along with the URL itself.
Update:
Here is what my code looks like:
import requests
import json
url = ("api.example.com/v1.11/user?id=123456")
header = {"Authorization": "hibnn:11111:77788777YT666:CAL1"}
response = requests.get(url, headers=header)
print(response.json)
However, I am getting a response [401] error. I think I am probably doing something wrong with my headers, can anyone help?
You may pass headers as keyword argument to the get method.
>>> url = 'https://api.github.com/some/endpoint'
>>> headers = {'user-agent': 'my-app/0.0.1'}
>>> r = requests.get(url, headers=headers)
Reference
I am trying to download my data by using Fitbit API. I have figured out how to obtain a certain day's data, which is good. And here is the curl command I used:
curl -i -H "Authorization: Bearer (here goes a very long token)" https://api.fitbit.com/1/user/-/activities/heart/date/2016-6-14/1d/1sec/time/00:00/23:59.json >> heart_rate_20160614.json
However, I would like to collect hundreds of days' data and I don't want to do that manually. So I think I could use a Python loop. I read some other topics like this one and this one but still don't know how to 'translate' these curl commands into python language by using urllib2.
I have tried this:
import urllib2
url = 'https://api.fitbit.com/1/user/-/activities/heart/date/today/1d/1sec/time/00:00/00:01.json'
data = '{Authorization: Bearer (here goes a very long token)}'
req = urllib2.Request(url,data)
f = urllib2.urlopen(req)
but the got an error says "HTTP Error 404: Not Found"
So what is the correct way to 'translate' this curl command to python language? Thanks!
The problem comes from the construction of the Request object : by default, the second parameter is the data that you want to pass along with the request. Instead, you have to specify that you want to pass headers. This is the correct way to do it :
import urllib2
url = 'https://api.fitbit.com/1/user/-/activities/heart/date/2016-6-14/1d/1sec/time/00:00/23:59.json'
hdr = {'Authorization': 'Bearer (token)'}
req = urllib2.Request(url,headers=hdr)
f = urllib2.urlopen(req)
This wields a 401 on my side, but should work with your token.
You can have more informations on urllib2 (and the Request class) here
However, I suggest you take a look at Requests, which is in my opinion easier to use, and very well documented.
Hope it'll be helpful.
You can use the excellent lib requests which is much easier to use than urllib, in my opinion.
First, pip install requests, then in your interpreter:
import requests
response = requests.get(url='https://api.fitbit.com/1/user/-/activities/heart/date/2016-6-14/1d/1sec/time/00:00/23:59.json', headers={'Authorization':'Bearer <TOKEN>'})
if response.ok:
print response.content
else:
print "error", response.content
From here you can easily get the response content via response.content or response.json() if it's a JSON, and write it to a file.
rocksteady's solution worked
He did originally refer to dictionaries. But the following code to send the JSON string also worked wonders using requests:
import requests
headers = {
'Authorization': app_token
}
url = api_url + "/b2api/v1/b2_get_upload_url"
content = json.dumps({'bucketId': bucket_id})
r = requests.post(url, data = content, headers = headers)
I'm working with an API that requires me to send JSON as a POST request to get results. Problem is that Python 3 won't allow me to do this.
The following Python 2 code works fine, in fact it's the official sample:
request = urllib2.Request(
api_url +'/b2api/v1/b2_get_upload_url',
json.dumps({ 'bucketId' : bucket_id }),
headers = { 'Authorization': account_authorization_token }
)
response = urllib2.urlopen(request)
However, using this code in Python 3 only makes it complain about data being invalid:
import json
from urllib.request import Request, urlopen
from urllib.parse import urlencode
# -! Irrelevant code has been cut out !-
headers = {
'Authorization': app_token
}
url = api_url + "/b2api/v1/b2_get_upload_url"
# Tested both with encode and without
content = json.dumps({'bucketId': bucket_id}).encode('utf-8')
request = Request(
url=url,
data=content,
headers=headers
)
response = urlopen(req)
I've tried doing urlencode(), like you're supposed to. But this returns a 400 status code from the web server, because it's expecting pure JSON. Even if the pure JSON data is invalid, I need to somehow force Python into sending it.
EDIT: As requested, here are the errors I get. Since this is a flask application, here's a screenshot of the debugger:
Screenshot
Adding .encode('utf-8') gives me an "Expected string or buffer" error
EDIT 2: Screenshot of the debugger with .encode('utf-8') added
Since I have a similar application running, but the client still was missing, I tried it myself.
The server which is running is from the following exercise:
Miguel Grinberg - designing a restful API using Flask
That's why it uses authentication.
But the interesting part: Using requests you can leave the dictionary as it is.
Look at this:
username = 'miguel'
password = 'python'
import requests
content = {"title":"Read a book"}
request = requests.get("http://127.0.0.1:5000/api/v1.0/projects", auth=(username, password), params=content)
print request.text
It seems to work :)
Update 1:
POST requests are done using requests.post(...)
This here describes it well : python requests
Update 2:
In order to complete the answer:
requests.post("http://127.0.0.1:5000/api/v1.0/projects", json=content)
sends the json-string.
json is a valid parameter of the request and internally uses json.dumps()...