I am attempting to upload a CSV file to an API (which gives me very little error info) using python requests library.
(I'm running Python 3.5 and using requests version 2.18.4 on OS X 10.11.6)
This curl command in terminal works fine: curl -F 'file=#/path/to/file.csv' myurl.com/upload -H "Authorization: TOKEN sometoken"
A multipart/form-data POST request from Postman also works, but I can't seem to make it work with the python requests library.
I've tried many variations of this request:
import requests
headers = {'Authorization': 'TOKEN sometoken', 'Content-Type': 'multipart/form-data'}
with open(file_path, 'rb') as f:
r = requests.post(myurl, headers=headers, data=f)
## I've also tried data={"file": f}
I get a status code of 200, but the response is {"success": "false"} (frustratingly unhelpful).
What am I missing in the python request compared to the curl request?
EDIT: it seems that the -F flag for the curl command emulates an HTML form that has been submitted...is there a way to do this with requests?
In your curl code you're using the -F parameter, which submits the data as a multipart message - in other words you're uploading a file.
With requests you can post files with the files parameter. An example:
import requests
headers = {'Authorization': 'TOKEN sometoken'}
data = {'file': open(file_path, 'rb')}
r = requests.post(myurl, headers=headers, files=data)
Note that requests creates the Content-Type header automatically, based on the submitted data.
The python open() function returns a file object. This is not what you want to send to the API.
Instead, you can use:
with open(file_path, 'rb') as f:
r = requests.post(myurl, headers=headers, data=f.read())
Syntax taken from here.
Related
when i make the following request:
curl -X POST http://localhost:8080/endpoint \
-F "file=filepath/example.zip" \
-H "Content-Type: multipart/form-data"
everything works fine.
However, when i try to do the same in python:
import requests
url = "http://localhost:8080/endpoint"
files = {"file": open("example.zip", "rb")}
headers = {"Content-Type": "multipart/form-data"}
r = requests.post(url, files=files, headers=headers)
the api gives me an error. are these two code snippets not equivalent? thanks
the issue was this:
multipart data POST using python requests: no multipart boundary was found
it seems as though setting
headers = {"Content-Type": "multipart/form-data"}
caused the issue. i removed it and it works
I have a simple python 3 script that send a post request to delete a project in SonarQube. While I am keep getting in my python script, a simple curl commands works... any idea what is wrong with my python script?
import requests
headers = {
'Authorization': 'Basic YWRtaW46YWRtaW4=',
}
files = [
('key', 'com.eclipseoptions.viewserver:viewserver:feature_VS-313-add-an-instruction-event-and-view'),
]
r = requests.post('http://devsonar/api/projects/delete', headers=headers, files=files)
print(r)
The following curl command works fine:
curl -X POST -H "Authorization: Basic YWRtaW46YWRtaW4=" -F "key=com.eclipseoptions.viewserver:viewserver:feature_VS-313-add-an-instruction-event-and-view" "http://devsonar/api/projects/delete"
Python requests is really a good library. Files option in post is used to upload file and I don't think com.eclipseoptions.viewserver:viewserver:feature_VS-313-add-an-instruction-event-and-view is a file if so, you have to read the file in binary mode and then send it like files = {'key': open(filename, 'rb')}. so the code should be:
import requests
files = {'key': open(filename, 'rb')}
headers = {'Authorization': 'Basic YWRtaW46YWRtaW4='}
response=requests.post(url,files=files)
check this for details on uploading files using requests library in python.
If it is not a file you can send the payload directly as a dictionary like this:
import requests
headers = {'Authorization': 'Basic YWRtaW46YWRtaW4='}
data = {'key': 'com.eclipseoptions.viewserver:viewserver:feature_VS-313-add-an-instruction-event-and-view'}
response=requests.post(url,data=data,headers=headers)
check this for details on sending payload.
You should have used data, not files, as an input to the python script, this should work:
import requests
headers = {
'Authorization': 'Basic YWRtaW46YWRtaW4=',
}
files = [
('key', 'com.eclipseoptions.viewserver:viewserver:feature_VS-313-add-an-instruction-event-and-view'),
]
r = requests.post('http://devsonar/api/projects/delete', headers=headers, data=files)
There is a cURL example in section Triggering a Build of TeamCity 9.x Documentation:
curl -v -u user:password http://teamcity.server.url:8111/app/rest/buildQueue --request POST --header "Content-Type:application/xml" --data-binary #build.xml
I'd like to know how to convert it into an equivalent Python script (using POST request from the requests module)?
BTW, I tried the following Python script but got such a response code 400 (Bad Request):
url = "http://myteamcity.com:8111/httpAuth/app/rest/buildQueue/"
headers = {'Content-Type': 'application/json'}
data = json.dumps({'buildTypeId': 'MyTestBuild'})
r = requests.post(url, headers=headers, data=data, auth=("username", "password"), timeout=10)
print "r = ", r
>> r = <Response [400]>
If change the Content-Type in headers into Accept, got another response code 415 (Unsupported Media Type):
headers = {'Accept': 'application/json'}
>> r = <Response [415]>
The documentation for triggering a build shows you need to send XML, not JSON:
<build>
<buildType id="buildConfID"/>
</build>
The TeamCity REST API is a bit of a mixed bag; some methods accept both XML and JSON, some only accept XML. This is one of those latter methods. They'll respond with either XML or JSON, based on what you set the Accept header to.
Send the above with your required build ID; for an XML document that simply you could use templating:
from xml.sax.saxutils import quoteattr
template = '<build><buildType id={id}/></build>'
url = "http://myteamcity.com:8111/httpAuth/app/rest/buildQueue/"
headers = {'Content-Type': 'application/xml'}
build_id = 'MyTestBuild'
data = template.format(id=quoteattr(build_id))
r = requests.post(url, headers=headers, data=data, auth=("username", "password"), timeout=10)
Note that I used the xml.sax.saxutils.quotattr() function to make sure the value of build_id is properly quoted for inclusion as a XML attribute.
This'll produce XML; add 'Accept': 'application/json' to the headers dictionary if you want to process a JSON response.
FYI json request works in TeamCity 10.
Since this question was written & answered, a contemporary OSS alternative now exists: pyteamcity.
Pip command to install (or add pyteamcity to requirements.txt, etc.)
pip install pyteamcity
Code:
from pyteamcity import TeamCity
tc = TeamCity('username', 'password', 'server', 'port')
result = tc.trigger_build('build_id')
print(f'Build triggered. Web URL: {result['webUrl']}')
I'm having an issue converting a working cURL call to an internal API to a python requests call.
Here's the working cURL call:
curl -k -H 'Authorization:Token token=12345' 'https://server.domain.com/api?query=query'
I then attempted to convert that call into a working python requests script here:
#!/usr/bin/env python
import requests
url = 'https://server.domain.com/api?query=query'
headers = {'Authorization': 'Token token=12345'}
r = requests.get(url, headers=headers, verify=False)
print r
I get a HTTP 401 or 500 error depending on how I change the headers variable around. What I do not understand is how my python request is any different then the cURL request. They are both being run from the same server, as the same user.
Any help would be appreciated
Hard to say without knowing your api, but you may have a redirect that curl is honoring that requests is not (or at least isn't send the headers on redirect).
Try using a session object to ensure all requests (and redirects) have your header.
#!/usr/bin/env python
import requests
url = 'https://server.domain.com/api?query=query'
headers = {'Authorization': 'Token token=12345'}
#start a session
s = requests.Session()
#add headers to session
s.headers.update(headers)
#use session to perform a GET request.
r = s.get(url)
print r
I figured it out, it turns out I had to specify the "accept" header value, the working script looks like this:
#!/usr/bin/env python
import requests
url = 'https://server.domain.com/api?query=query'
headers = {'Accept': 'application/app.app.v2+json', 'Authorization': 'Token token=12345'}
r = requests.get(url, headers=headers, verify=False)
print r.json()
I have been trying to use python to get data from an API - which I am able to access manually like below
curl --location --compressed --header “Authorization: Bearer [ACCESS_TOKEN]” [STREAM_URL]
How do I go about using it in python - I have read examples it says use requestsbut how to handle the Access_Token part in python
Any help would be appreciated?
Regards
VB
Just define your headers like you do in your curl example. In this case though, it's a dict. Assuming you want to use requests you would do something like this:
import requests
url = [STREAM_URL]
headers = {"Authorization": "Bearer [ACCESS TOKEN]"}
r = requests.get(url, headers=headers)
print(r.text)
There's no different what to use. urllib come with the python and if you don't like to download additional package, use urllib.
As OP mentioned --compressed in curl command, we should do the same to get gzipped content:
Edited ars answer:
import sys
if sys.version_info.major == 3:
import urllib.request as urllib2
else:
import urllib2
from StringIO import StringIO
import gzip
request = urllib2.Request('http://example.com/')
request.add_header('Accept-encoding', 'gzip')
request.add_header('Authorization', 'Bearer [ACCESS_TOKEN]')
response = urllib2.urlopen(request)
if response.info().get('Content-Encoding') == 'gzip':
buf = StringIO( response.read())
f = gzip.GzipFile(fileobj=buf)
data = f.read()
Does python urllib2 automatically uncompress gzip data fetched from webpage?