We Keep Coding

Using Python requests for automatic file uploading to Flask environment

I am trying to setup a script where I upload a file (now using the python requests library) to a Flask environment that runs inside of a Docker (-compose) container. The python script is ran in the hypervisor, and I force it to use the same version of python (3.6) as well. I am able to get responses from the server, but the file I upload is 12Kb, and the file the Flask container receives is 2Kb, and I have no clue what is going wrong.
It seems like when I use WireShark to capture the tcp stream, I receive a 2Kb file as well, so my guess the requests library applies some compression, but I can not seem to find any documentation about this happening.
I have tried to replace the file tuple in the sending code with solely the file handle, but this seemed to have no effect. Sending files of a different size results in a different filesize in Flask / Docker.
Sending a string instead of the filehandler ("1234567890") results in the filesize being as big as the string length (10 bytes).
Replacing the file opening method from rb to r results in a UnicodeDecodeError: 'ascii' codec can't decode byte 0xdf in position 14: ordinal not in range(128) raise inside requests -> encodings.
Hypervisor: send.py
import requests
with open('file.docx', 'rb') as f:
url = 'http://localhost:8081/file'
r = requests.post(url, files={'file': ('file', f, 'multipart/form-data')})
print(r.text)
Flask: file.py
#app.route('/file', methods=['POST'])
def parse_from_post():
file = request.files['file']
fn = secure_filename("file.docx") # did this manually instead of getting it from the request for testing reasons
folder = "/app/files/"
fl = os.path.join(folder, fn)
# Removes old file
if os.path.exists(fl):
os.remove(fl)
file.save(fl)
return ""

The problem lies with the filesize, which python requests does not take care of directly. I used the MultipartEncoder from the requests_toolbelt package, to encapsulate the file instead of directly plugging in the file in the requests post call.
Hypervisor: send.py
import requests
from requests_toolbelt import MultipartEncoder
with open('file.docx', 'rb') as f:
url = 'http://localhost:8081/file'
m = MultipartEncoder(fields={
"file": ("file.docx", f)
})
r = requests.post(url, data=m, headers={'Content-Type': m.content_type})
print(r.text)
I actually found this result from another post on SO, see the first comment on the question linking to https://toolbelt.readthedocs.io/....

How to download file using python, requests module

I need to open the page automatically and download the file returned by the server
I have a simple code to open the page and download the content. I am also pulling the headers so I know the name of the returned file. below is the code
downloadPageRequest = self.reqSession.get( self.url_file ,stream=True)
headers = downloadPageRequest.headers
if 'content-disposition' in headers:
file_name = re.findall("filename=(.+)", headers['content-disposition'])
that's what I got, it returns an array with the filename, but now I am stuck and have no idea how to open and go through returned excel file
this has to be done using requests, that's why i cannot use any other method (e.g selenium)
will be thankful for your support

Python FileNotFoundError when reading a public json file

I have a public viewable JSON file that is hosted on s3. I can access the file directly by clicking on the public url to the s3 object and view the JSON in my browser fine. Anyone on the internet can view the file easily.
Yet with the below code is ran in Python (using Lambda connected to an API trigger) I get [Errno 2] No such file or directory: as the errorMessage, and FileNotFoundError as the errorType.
def readLeetDictionary(self):
jsonfile = 'https://s3.amazonaws.com/url/to/public/facing/file/data.json'
with open(jsonfile, 'r') as data_file:
self.data = json.load(data_file)
What am I missing here? Since the file is a publicly viewable JSON file I would assume I wouldn't be forced to use boto3 library and formally handshake to the file in order to read the file (with object = s3.get_object(Bucket=bucket_names,Key=object_name) for example) - would I?

The conde you need should be something like:
import urllib, json
def readLeetDictionary():
jsonfile = 'https://s3.amazonaws.com/url/to/public/facing/file/data.json'
response = urllib.urlopen(jsonfile)
data = json.loads(response.read())
print data
Please feel free to ask further or explain if this does not suit you.

Passing Binary file over HTTP POST

I have a local python file that decodes binary files. This python file first reads from the file, opens it as binary and then saves it in a buffer and interprets it. Reading it is simply:
with open(filepath, 'rb') as f:
buff = f.read()
read_all(buff)
This works fine locally. Now I'd like to setup a Azure Python job where I can send the file, approx. 100kb, over a HTTP POST and then read the interpreted meta data which my original python script does well.
I've first removed the read function so that I'll now work with the buffer only.
In my Azure Python Job I have the following, triggered by a HttpRequest
my_data = reader.read_file(req.get_body())
To test my sending I've tried the following in python
import requests
url = 'http://localhost:7071/api/HttpTrigger'
files = {'file': open('test', 'rb')}
with open('test', 'rb') as f:
buff = f.read()
r = requests.post(url, files=files) #Try using files
r = requests.post(url, data=buff) #Try using data
I've also tried in Postman adding the file to the body as a binary and setting the headers to application/octet-stream
All this doesn't send the binary file the same way as the original f.read() did. So I'm getting a wrong interpretation of the binary file.
What is file.read doing differently to how I'm sending it over as a HTTP Body message?
Printing out the first line from the local python read file gives.
b'\n\n\xfe\xfe\x00\x00\x00\x00\\\x18,A\x18\x00\x00\x00(\x00\x00\x00\x1f\x00\x00\
Whereas printing it out at the req.get_body() shows me
b'\n\n\xef\xbf\xbd\xef\xbf\xbd\x00\x00\x00\x00\\\x18,A\x18\x00\x00\x00(\x00\x00\x00\x1f\x00\
So something is clearly wrong. Any help why this could be different?
Thanks
EDIT:
I've implemented a similar function in Flask and it works well.
The code in flask is simply grabbing the file from a POST. No encoding/decoding.
if request.method == 'POST':
f = request.files['file']
#f.save(secure_filename(f.filename))
my_data = reader.read_file(f.read())
Why is the Azure Function different?

You can try UTF-16 to decode and do the further action in your code.
Here is the code for that:
with open(path_to_file,'rb') as f:
contents = f.read()
contents = contents.rstrip("\n").decode("utf-16")
Basically after doing re.get_body, perform the below operation:
contents = contents.rstrip("\n").decode("utf-16")
See if it gives you the same output as your receive in local python file.
Hope it helps.

Download files using Python

I'm trying to download a few files using roboBrowser, URLLIB or any other python library, but I couldn't find a way to make it work.
Basically, I have a form which retrieves a .CSV file when is submitted, but I couldn't find any way to start this download.
I have submitted the form using RoboBrowser and URLLIB post but I couldn't reach the file
Form = browser.get_form(action=re.compile(r'downloadForm'))
Form ["d_screen_file"].value = "1"
browser.submit_form(Form , submit=programForm['download'])
or
action = browser.find('form', id='fx_form').get('action')
requests.post(action)
There is another way to submit this form/make this requisition to engage this download?

I figure out how to make it work:
Using requests I do it a post with stream=True
f = session.post(FormRequest, data=search_data, stream=True)
After that, I create a CSV file to receive this data and use a for loop to parse the data using iter_content and write in the file
with open("file.csv", 'wb') as s:
for chunk in f.iter_content(chunk_size=1024):
s.write(chunk)