We Keep Coding

Passing a remote file to a http post request in python

I have a rest api that offers an upload file post functionality. To test this, I simply did this on my python test code:
fo = open('file.zip', 'rb')
rb_file = {'confile': ('file.zip', fo, 'multipart/form-data')}
resp = requests.post(url,files=rb_file)
fo.close()
This request returns a uuid response {ce9f2d23-8ecd-4c60-9d31-aef0be103d44} that is needed for initiating another post run for a scan.
From Swagger, manually passing this uuid to the scan post request generates the following curl:
curl -X POST "http://....../scan" -H "Content-Type: multipart/form-data" -F "FilID=ce9f2d23-8ecd-4c60-9d31-aef0be103d44"
My question is how to convert this curl to a python request code noting that I no longer have the file on my local machine. The server seems to be expecting a -F rather than a param. How do I pass a file that doesn't exist on my local machine to http request in this case? All I have is the filID. I tried running as param but that doesn't find the resource.

try this !
import requests
headers = {
'Content-Type': 'multipart/form-data',
}
files = {
'FilID': (None, 'ce9f2d23-8ecd-4c60-9d31-aef0be103d44'),
}
response = requests.post('http://....../scan', headers=headers, files=files)

Replacing curl with python requests equivalent to POST a file to a server

After writing a file with the snippet below
with open("temp.trig", "wb") as f:
f.write(data)
I use curl to load it into the server
curl -X POST -F file=#"temp.trig" -H "Accept: application/json" http://localhost:8081/demo/upload
which works fine.
I am trying to replace the curl with python requests, as follows:
with open("temp.trig", "rb") as f:
result = requests.post("http://localhost:8081/demo/upload", files={'file': f},
headers = {"Accept": "application/json"})
which attempted to follow the curl as closely as possible. This code results in an error 500 from the server. I suspect it must be something related to the request, because the same server is ok via `curl. Any ideas?

There probably is nothing wrong with your python script.
Differences I've noticed between curl and requests are the following:
obviously, User-Agent headers are different — curl/7.47.0 vs. python-requests/2.22.0
multipart boundary format in Content-Type header is different — ------------------------6debaa3504bbc177 in curl vs. c1e9f4f617de4d0dbdb48fcc5aab67e0 in requests
therefore Content-Length value will almost certainly be different
multipart/form-data format in body is slightly different — curl adds an extra line (Content-Type: text/plain) before file contents
So depending on your file format, server may not be able to parse requests HTTP request format.
I think the best solution for you now is to compare raw HTTP requests from curl and requests and find what differences are significant.
For example:
Open terminal
Launch netcat with nc -l -p 1234 command. This will listen to HTTP requests on localhost on port 1234 and output raw HTTP requests to terminal.
Send your curl request as it is to localhost:1234 in another tab
Execute your python script as it is using URL localhost:1234 in another tab
Compare raw requests from your netcat output

Here's my attempt:
import requests
headers = {
'Accept': 'application/json',
}
files = {
'file': ('temp.trig', open('temp.trig', 'rb')),
}
response = requests.post('http://localhost:8081/demo/upload', headers=headers, files=files)
In case this doesn't work we really need to read more data on the server side, as Ivan Vinogradov explained well.

Python 2.7 requests REST API call not working same as curl

I am trying to do a curl post query using requests python 2.7, however the API response differently using curl vs. requests lib.
The post query is pretty simple a file and name-value-pair data as the API params.
Below is the curl multipart post request:
curl -uadmin:blabla123 -X POST 127.0.0.1:8080/alfresco/api/-default-/public/alfresco/versions/1/nodes/6a0ab661-1c43-43ed-b07f-a564f6bcb5ca/children -F filedata=#file1.txt -F name=document__55;nodeType=content
The python 2.7 code is as below:
import requests
from requests.auth import HTTPBasicAuth, HTTPDigestAuth
from config import USER, PASSWD
def createDocument( documentFilename, documentMetadata, targetFolderNodeId):
'''
Uploads a file and its meta-data to the CMIS server under the specified
target folder
'''
with open(documentFilename, 'rb') as file:
files = {'file': file}
# createURL = 'http://127.0.0.1:8080/alfresco/api/-default-/public/alfresco/versions/1/nodes/{0}/children'.format( targetFolderNodeId )
createURL = 'http://127.0.0.1:8080/alfresco/api/-default-/public/alfresco/versions/1/nodes/6a0ab661-1c43-43ed-b07f-a564f6bcb5ca/children'
data = {
"name":"document__55",
"nodeType":"cm:content",
}
response = requests.post( createURL, data = data, files = files, auth=HTTPBasicAuth(USER, PASSWD) )
print(response)
print(response.json)
print(response.text)
createDocument('file1.txt', '', '')
Curl returns 200 http code but the script oddly returns 400.
Any help is much appreciate it.

At a first glace the reasons for that script failing vs curl working could be more than one:
nodeType=content in the curl, while "nodeType":"cm:content" in the script
there's an additional comma in the script's payload second line
you're not setting the Content-Type header as multipart/form-data in the script, which curl's -F option actually does
Also, the endpoint should provide a verbose error along with the 400 response. If it does, that could be helpful to identify the error cause.

in fact in the curl I have -F filedata=#file1.txt but in the script I have files = {'file': file} which is not the same so I just had to use files = {'filedata': file} :)
def createDocument( documentFilename, documentMetadata, targetFolderNodeId):
'''
Uploads a file and its meta-data to the CMIS server under the specified
target folder
'''
with open(documentFilename, 'rb') as file:
files = {'filedata': file}
createURL = 'http://127.0.0.1:8080/alfresco/api/-default-/public/alfresco/versions/1/nodes/{0}/children'.format( targetFolderNodeId )
data = {
"name":"document__77",
"nodeType":"cm:content"
}
response = requests.post( createURL, data = data, files = files, auth=HTTPBasicAuth(USER, PASSWD) )
print(response)
print(response.json)
print(response.text)
print(response.headers)
# print([i for i in dir(response) if 'header' in i])

unable to post file+data using python-requests

I'm able to post file using curl
curl -X POST -i -F name='barca' -F country='spain' -F
file=#/home/messi/Desktop/barca.png 'http://localhost:8080/new_org/hel/concerts'
Which I can get (file) as
curl -X GET -H 'Accept: image/png' 'http://localhost:8080/new_org/hel/concerts/<id or name of entity>'
But when I tried same thing using requests.post, I got error. Does anybody know why this happen. (Post Error encounter when file pointer is not at last, but when file pointer is at last, I got response 200 but file is not posted)
import requests
url = 'http://localhost:8080/new_org/hel/concerts'
file = dict(file=open('/home/messi/Desktop/barca.png', 'rb'))
data = dict(name='barca', country='spain')
response = requests.post(url, files=file, data=data)
Error: (from usergrid) with response code: 400
{u'duration': 0,
u'error': u'illegal_argument',
u'error_description': u'value is null',
u'exception': u'java.lang.IllegalArgumentException',
u'timestamp': 1448330119021}
https://github.com/apache/usergrid

I believe the problem is that Python is not sending a content-type field for the image that you are sending. I traced through the Usergrid code using a debugger and saw that curl is sending the the content-type for the image and Python is not.
I was able to get this exact code to work on my local Usergrid:
import requests
url = 'http://10.1.1.161:8080/test-organization/test-app/photos/'
files = { 'file': ('13.jpg', open('/Users/dave/Downloads/13.jpg', 'rb'), 'image/jpeg')}
data = dict(name='barca', country='spain')
response = requests.post(url, files=files, data=data)
It is possible that Waken Meng's answer did not work because of the syntax of the files variable, but I'm no Python expert.

I met a problem before when i try to upload image files. Then I read the doc and do this part:
You can set the filename, content_type and headers explicitly:
Here is how I define the file_data:
file_data = [('pic', ('test.png', open('test.png'), 'image/png'))]
r = requests.post(url, files=file_data)
file_data should be a a list: [(param_name, (file_name, file, content_type))]
This works for me, hope can help you.

How to convert cURL in Python requests to change Jenkins config.xml file

I have this cURL:
curl -X POST http://user:pass#blabla.com:8080/job/myproject/config.xml --data-binary "#new_config.xml"
I am basically trying to set a new config for a Jenkins installation by changing the pre-existing config.xml file.
I am trying to convert it to something like this in order to use it more flexibly in my code:
url = "http://host:8080/job/myproject/config.xml"
auth = ('user','pass')
payload = {"--data-binary": "#new_config.xml"}
headers = {"Content-Type" : "application/xml"}
r = requests.post(url, auth=auth, data=payload, headers=headers)
I know that I am using incorrectly the payload and the headers.How should I change them?
I run it and I take a 500 responce code.
I read this post , but I am struggling to apply it in my case.

The --data-binary switch means: post the command line argument as the whole POST body, without wrapping in multipart/form-data or application/x-www-form-encoding containers. # tells curl to load the data from a filename; new_config.xml in this case.
You'll need to open the file object to send the contents as the data argument:
url = "http://host:8080/job/myproject/config.xml"
auth = ('user','pass')
headers = {"Content-Type" : "application/xml"}
with open('new_config.xml', 'rb') as payload:
r = requests.post(url, auth=auth, data=payload, headers=headers)
Note that I pass the file object directly into requests; the data will then be read and pushed to the HTTP socket, streaming the data efficiently.