I need the bot to download the file from the link. I am using this function:
def download(url, filename):
get_response = requests.get(url,stream=True)
file_name = filename
with open(file_name, 'wb') as f:
for chunk in get_response.iter_content(chunk_size=1024):
if chunk: # filter out keep-alive new chunks
f.write(chunk)
And then I send it to the user using this:
with open(music, 'rb') as music_file:
await bot.send_audio(message.chat.id, music_file)
Everything works great on my computer. But when I run the bot on Heroku, it constantly gives this error:
aiogram.utils.exceptions.BadRequest: File must be non-empty
I tried to add delays, tried to check if the file was downloaded, but nothing helped. Can anyone help?
Related
What I need to do is to write some messages on a .txt file, close it and send it to a server. This happens in a infinite loop, so the code should look more or less like this:
from requests_toolbelt.multipart.encoder import MultipartEncoder
num = 0
while True:
num += 1
filename = f"example{num}.txt"
with open(filename, "w") as f:
f.write("Hello")
f.close()
mp_encoder = MultipartEncoder(
fields={
'file': ("file", open(filename, 'rb'), 'text/plain')
}
)
r = requests.post("my_url/save_file", data=mp_encoder, headers=my_headers)
time.sleep(10)
The post works if the file is created manually inside my working directory, but if I try to create it and write on it through code, I receive this response message:
500 - Internal Server Error
System.IO.IOException: Unexpected end of Stream, the content may have already been read by another component.
I don't see the file appearing in the project window of PyCharm...I even used time.sleep(10) because at first, I thought it could be a time-related problem, but I didn't solve the problem. In fact, the file appears in my working directory only when I stop the code, so it seems the file is held by the program even after I explicitly called f.close(): I know the with function should take care of closing files, but it didn't look like that so I tried to add a close() to understand if that was the problem (spoiler: it was not)
I solved the problem by using another file
with open(filename, "r") as firstfile, open("new.txt", "a+") as secondfile:
secondfile.write(firstfile.read())
with open(filename, 'w'):
pass
r = requests.post("my_url/save_file", data=mp_encoder, headers=my_headers)
if r.status_code == requests.codes.ok:
os.remove("new.txt")
else:
print("File not saved")
I make a copy of the file, empty the original file to save space and send the copy to the server (and then delete the copy). Looks like the problem was that the original file was held open by the Python logging module
Firstly, can you change open(f, 'rb') to open("example.txt", 'rb'). In open, you should be passing file name not a closed file pointer.
Also, you can use os.path.abspath to show the location to know where file is written.
import os
os.path.abspath('.')
Third point, when you are using with context manager to open a file, you don't close the file. The context manger supposed to do it.
with open("example.txt", "w") as f:
f.write("Hello")
I have Django project on Cloud Run. When I download small file from page which has below code.
def download_view(request,pk):
file_path = f'media/{pk}.mp3'
name = f'{pk}.mp3'
with open(file_path, 'rb') as f:
response = HttpResponse(f.read(), content_type='audio/wav')
response['Content-Disposition'] = f'attachment; filename={name}'
return response
It's works fine. However, when I download a file (50MB). I got this picture's error.
Cloud run's log is like this. I couldn't find any log of traceback.
2021-05-06 12:00:35.668 JSTGET500606 B66 msChrome 72 https://***/download/mp3/2500762/
2021-05-06 11:49:49.037 JSTGET500606 B61 msChrome 72 https://***/download/mp3/2500645/
I'm not sure. Is this error related with download error.
2021-05-06 16:48:32.570 JSTResponse size was too large. Please consider reducing response size.
I think this is upload file size error. So this is not related with this subject of download error.
When I run Django at local, then download same 50MB file. I can download it. I think this download error related with Cloud run. It's stop after request/response. So I think this error coused by Cloud Run. Which was stoped, when I'm still downloading file.
I don't know how to solve this download error. If you have any solution, please help me!
The Cloud Run HTTP request/response size is limited to 32Mb. Use a multipart/form-data to send chunks of your big file and not the whole file directly.
Thank you #guillaume blaquiere! I solved download error. I post my code for othres.
def _file_iterator(file, chunk_size=512):
with open(file, 'rb') as f:
while True:
c = f.read(chunk_size)
if c:
yield c
else:
break
def download_view(request,pk):
file_path = f'media/{pk}.mp3'
file_name = f'{pk}.mp3'
response = StreamingHttpResponse(_file_iterator(file_path))
response['Content-Type'] = 'audio/mpeg'
response['Content-Disposition'] = f'attachment;filename="{file_name}"'
return response
I think StreamingHttpResponse is key point of this problem. It's return big file by chunks. It dose not over Cloud Run's limit.
When I used multipart/form-data for Content-Type, I could download file. But it's couldn't open on smart phone, because It couldn't select application. When I download on PC, it's can't show audio file icon. We should select exact content type.
I have this code for server
#app.route('/get', methods=['GET'])
def get():
return send_file("token.jpg", attachment_filename=("token.jpg"), mimetype='image/jpg')
and this code for getting response
r = requests.get(url + '/get')
And i need to save file from response to hard drive. But i cant use r.files. What i need to do in these situation?
Assuming the get request is valid. You can use use Python's built in function open, to open a file in binary mode and write the returned content to disk. Example below.
file_content = requests.get('http://yoururl/get')
save_file = open("sample_image.png", "wb")
save_file.write(file_content.content)
save_file.close()
As you can see, to write the image to disk, we use open, and write the returned content to 'sample_image.png'. Since your server-side code seems to be returning only one file, the example above should work for you.
You can set the stream parameter and extract the filename from the HTTP headers. Then the raw data from the undecoded body can be read and saved chunk by chunk.
import os
import re
import requests
resp = requests.get('http://127.0.0.1:5000/get', stream=True)
name = re.findall('filename=(.+)', resp.headers['Content-Disposition'])[0]
dest = os.path.join(os.path.expanduser('~'), name)
with open(dest, 'wb') as fp:
while True:
chunk = resp.raw.read(1024)
if not chunk: break
fp.write(chunk)
I am trying to convert json to csv and download a file from my flask application. The function does not work correctly, I always get the same csv, even if I delete the json file. Why?
button:
Download
My method:
#app.route("/download/<file_id>")
def get_csv(file_id):
try:
file_id = f"{file_id}"
filename_jsonl = f"{file_id}.jsonl"
filename_csv = f"{file_id}.csv"
file_id = ''
with open(filename_jsonl, 'r') as f:
for line in f.read():
file_id += line
file_id = [json.loads(item + '\n}') for item in file_id.split('}\n')[0:-1]]
with open(filename_csv, 'a') as f:
writer = csv.DictWriter(f, file_id[0].keys(), delimiter=";")
writer.writeheader()
for profile in file_id:
writer.writerow(profile)
return send_from_directory(directory='', filename=filename_csv, as_attachment=True)
except FileNotFoundError:
abort(404)
The problem you are having is that the first generated file has been cached.
Official documentation says that send_from_directory() send a file from a given directory with send_file(). send_file() sets the cache_timeout option.
You must configure this option to disable caching, like this:
return send_from_directory(directory='', filename=filename_csv, as_attachment=True, cache_timeout=0)
#app.route('/download')
def download():
return send_from_directory('static', 'files/cheat_sheet.pdf')
Note: First parameter give it the directory name like static if your file is inside static (the file only could be in the project directory),
and for the second parameter write the right path of the file. The file will be automatically downloaded, if the route got download.
I have a video in a url, that I want download it using Python.
The problem here is that when I execute the script and download it, the final file just have 1 kb, it's like never start the process of download.
I tried with this solution that I saw in https://stackoverflow.com/a/16696317/5280246:
url_video = "https://abiu-tree.fruithosted.net/dash/m/cdtsqmlbpkbmmddq~1504839971~190.52.0.0~w7tv1per/init-a1.mp4"
rsp = requests.get(url_video, stream=True)
print("Downloading video...")
with open("video_test_10.mp4",'wb') as outfile:
for chunk in rsp.iter_content(chunk_size=1024):
if chunk:
outfile.write(chunk)
rsp.close()
Too I tried like this:
url_video = "https://abiu-tree.fruithosted.net/dash/m/cdtsqmlbpkbmmddq~1504839971~190.52.0.0~w7tv1per/init-a1.mp4"
rsp = requests.get(url_video)
with open("out.mp4",'wb') as f:
f.write(rsp.content)
I tried too with:
urllib.request.retrieve(url_video, "out.mp4")