I have the following cherrypy function. The function suppose the response with a CSV file. However, It doesn't works, and i get the following response: Something went wrong here List(UnacceptedResponseContentTypeRejection(WrappedArray(text/csv))).
#cherrypy.expose
def myFunction(self, id, a):
url = "clusters/downAsCSV?id=" + id + "&a=" + a + "&csv=true"
htmlText = self.general_url(url)
cherrypy.response.headers['Content-Type'] = 'text/csv'
return htmlText
When i paste the same URL into the browser it does works and the CSV been downloaded into the client.
What could be the reason?
Related
I have a request for you, can you please help me with one question.
How can I send a pdf file via api.
It starts like this, they send me the base64 format and I accept it and do the decoding.
Then I need to send this file to another endpoint, but I just can’t put it there. Could you please help me
According to Postman, my file sits quietly and works as it should. The picture shows that it takes the form-date.
def sendDocs(photoBack):
try:
headers = {'Content-Type':'multipart/form-data;',
'bsauth': 'key'}
import base64
decodedData = base64.b64decode((photoBack))
pdfFile = open('photoBack.pdf', 'rb+')
pdfFile.write(decodedData)
pdfFile.close()
response = HttpResponse(pdfFile, content_type='application/pdf')
response['Content-Disposition'] = 'attachment; filename="photoBack.pdf"'
url = f'https://file.shinhanfinance.kz/files/shinhanfinance/add?client={1013246509}'
files = {"file":response}
firstPost1 = requests.post(url,data =response,headers=headers)
print(Response(firstPost1))
return Response({firstPost1})
except:
return Response({'bruh what wrong ?'})
Here my code
def sendDocs(photoBack):
try:
headers = {'Content-Type':'multipart/form-data;',
'bsauth': 'key'}
import base64
decodedData = base64.b64decode((photoBack))
pdfFile = open('photoBack.pdf', 'rb+')
pdfFile.write(decodedData)
pdfFile.close()
response = HttpResponse(pdfFile, content_type='application/pdf')
response['Content-Disposition'] = 'attachment; filename="photoBack.pdf"'
url = f'https://file.shinhanfinance.kz/files/shinhanfinance/add?client={1013246509}'
files = {"file":response}
firstPost1 = requests.post(url,data =response,headers=headers)
print(Response(firstPost1))
return Response({firstPost1})
except:
return Response({'bruh what wrong ?'})
You can try to use decodedData, and put it into file parameters.
firstPost1 = requests.post(url, files={"file": decodedData}, headers=headers)
I think here is your solution How to upload file with python requests?
What I have: I've a Flask web app deployed to Heroku's server, which consists of only one web process app.py. Here it is:
#importation
from flask import Flask, render_template, current_app, send_file, request, json, send_file
import os
#working functions
#json write
def json_write(dictionary):
with open("./json/info.json", "w+") as f:
json.dump(dictionary, f, indent=4)
#make file name
def make_file_name(name):
filename = "tube4u_"
for t in str(name):
if t.isalnum():
filename += t
filename += ".mp4"
return filename
#application initialisation
app=Flask(__name__)
#home
#app.route("/")
def home():
return render_template("index.html")
#processor
#app.route("/process/", methods=["GET"])
def process():
#get url
url = request.args["url"]
#import & initialisation
from pytube import YouTube
import pickle
json_dict = {}
try:
yt = YouTube(url)
except:
return "<h1>Invalid URL</h1>"
all_videos = yt.streams.filter(type="video", progressive=True)
json_dict["title"] = yt.title
json_dict["thumbnail"] = yt.thumbnail_url
json_dict["name"] = make_file_name(yt.title)
with open("./pickle/ytobj.pkl", "wb") as f:
pickle.dump(all_videos, f)
#videos with itag
json_dict["videos"] = [ {"itag": item.itag, "res": item.resolution} for item in all_videos]
json_write(json_dict)
return render_template("menu.html")
#download
#app.route("/download/", methods=["GET"])
def download():
import pickle
itag = int(request.args["itag"])
with open("./json/info.json") as f:
json_dict = json.load(f)
with open("./pickle/ytobj.pkl", "rb") as f:
all_videos = pickle.load(f)
video = all_videos.get_by_itag(itag)
video.download(output_path="./video", filename=f"{json_dict['name']}")
return render_template("thank.html")
#return video
#app.route("/video/", methods=["GET"])
def video():
filename = request.args["filename"]
return send_file(f"./video/{filename}", as_attachment=True)
#return json
#app.route("/json")
def fetchjson():
with open("./json/info.json") as f:
content = json.load(f)
return content
#get name
#app.route("/name")
def fetchname():
with open("./json/info.json") as f:
content = json.load(f)
return content
#app.route("/list")
def listall():
return f"{os.listdir('./video')}"
#running the app
if __name__ == "__main__":
app.run(debug=True)
How it works: here I made the app like that, whenever someone enter a URL and click Go then it creates a json file with the name info.json. after it gets everything properly it performs some task with the given URL reading from the file.
My problem:
Now the problem is, if I make a request of the web it will create a json with my given URL, suppose at the same time someone else make a request and enter a URL then server will lost my information and rewrite the json file with another client's given input URL my task will be performed with another's input url. It's really weird.
How to fix it? Like if there any way to create the info.json file on separate path for each client and gets deleted after work done?
There is a lot of ways in my point of view
When the server get client request then check if there is already a file.if there is already a file then add timestamp or add something else in the filename so the file will not be overwritten.
Ask the user file name and also add timestamp in the name and save it.
You can also use databases to store data of different clients .may be you can create login system and give every user an id and store data for every user in database accordingly.
So on...
You can see there is a lot of ways to solve this.
import json
import requests
def download_file(url):
r = requests.get(url)
filename = url.split('/')[-1]
with open(filename, 'wb') as f:
f.write(r.content)
api_url = 'https://api.fda.gov/download.json'
r = requests.get(api_url)
files = [file['file'] for file in json.loads(r.text)['results']['drug']['event']['partitions']]
count = 1
for file in files:
download_file(file)
print(f"{count}/{len(files)} downloaded!")
count += 1
This is the other code
import urllib.request, json
with urllib.request.urlopen("https://api.fda.gov/drug/label.json") as url:
data = json.loads(url.read().decode())
print(data)
The first code just downloads it. I wondering if theres a way to not have to download any of the 1000+ files and just display it, so the code can be used locally. While the second one prints the json in the terminal.
requests.get() and urllib.request.urlopen() both "download" the full response of the URL they are given.
If you do not want to "save" the file to disk, then remove the code that calls f.write()
More specifically,
import json
import requests
api_url = 'https://api.fda.gov/download.json'
r = requests.get(api_url)
files = [file['file'] for file in r.json()['results']['drug']['event']['partitions']]
total_files = len(files)
count = 0
for file in files:
print(requests.get(file).content)
print(f"{count+1}/{total_files} downloaded!")
count += 1
I have a following spider:
class Downloader(scrapy.Spider):
name = "sor_spider"
download_folder = FOLDER
def get_links(self):
df = pd.read_excel(LIST)
return df["Value"].loc
def start_requests(self):
urls = self.get_links()
for url in urls.iteritems():
index = {"index" : url[0]}
yield scrapy.Request(url=url[1], callback=self.download_file, errback=self.errback_httpbin, meta=index, dont_filter=True)
def download_file(self, response):
url = response.url
index = response.meta["index"]
content_type = response.headers['Content-Type']
download_path = os.path.join(self.download_folder, r"{}".format(str(index)))
with open(download_path, "wb") as f:
f.write(response.body)
yield LinkCheckerItem(index=response.meta["index"], url=url, code="downloaded")
def errback_httpbin(self, failure):
yield LinkCheckerItem(index=failure.request.meta["index"], url=failure.request.url, code="error")
It should:
read excel with links (LIST)
go to each link and download file to the FOLDER
log results in LinkCheckerItem(I am exporting it to csv)
That would normally work fine but my list contains files of different types - zip, pdf, doc etc.
These are the examples of links in my LIST:
https://disclosure.1prime.ru/Portal/GetDocument.aspx?emId=7805019624&docId=2c5fb68702294531afd03041e877ca84
http://www.e-disclosure.ru/portal/FileLoad.ashx?Fileid=1173293
http://www.e-disclosure.ru/portal/FileLoad.ashx?Fileid=1263289
https://disclosure.1prime.ru/Portal/GetDocument.aspx?emId=7805019624&docId=eb9f06d2b837401eba9c66c8bf5be813
http://e-disclosure.ru/portal/FileLoad.ashx?Fileid=952317
http://e-disclosure.ru/portal/FileLoad.ashx?Fileid=1042224
https://www.e-disclosure.ru/portal/FileLoad.ashx?Fileid=1160005
https://www.e-disclosure.ru/portal/FileLoad.ashx?Fileid=925955
https://www.e-disclosure.ru/portal/FileLoad.ashx?Fileid=1166563
http://npoimpuls.ru/templates/npoimpuls/material/documents/%D0%A1%D0%BF%D0%B8%D1%81%D0%BE%D0%BA%20%D0%B0%D1%84%D1%84%D0%B8%D0%BB%D0%B8%D1%80%D0%BE%D0%B2%D0%B0%D0%BD%D0%BD%D1%8B%D1%85%20%D0%BB%D0%B8%D1%86%20%D0%BD%D0%B0%2030.06.2016.pdf
http://нпоимпульс.рф/assets/download/sal30.09.2017.pdf
http://www.e-disclosure.ru/portal/FileLoad.ashx?Fileid=1166287
I would like it to save file with its original extension, whatever it is... Just like my browser when it opens an alert to save file.
I tried to use response.headers["Content-type"] to find out the type but in this case it's always application/octet-stream .
How could I do it?
You need to parse Content-Disposition header for the correct file name.
I would like to POST a .tgz file with the Python urllib2 library to a backend server. I can't use requests due to some licensing issues. There are some examples of file upload on stackoverflow but all relate to attaching a file in a form.
My code is the following but it unfortunately fails:
stats["random"] = "data"
statsFile = "mydata.json"
headersFile = "header-data.txt"
tarFile = "body.tgz"
headers = {}
#Some custom headers
headers["X-confidential"] = "Confidential"
headers["X-version"] = "2"
headers["Content-Type"] = "application/x-gtar"
#Create the json and txt files
with open(statsFile, 'w') as a, open(headersFile, 'w') as b:
json.dump(stats, a, indent=4)
for k,v in headers.items():
b.write(k+":"+v+"\n")
#Create a compressed file to send
tar = tarfile.open(tarFile, 'w:gz' )
for name in [statsFile,headersFile]:
tar.add(name)
tar.close()
#Read the binary data from the file
with open(tarFile, 'rb') as f:
content = f.read()
url = "http://www.myurl.com"
req = urllib2.Request(url, data=content, headers=headers)
response = urllib2.urlopen(req, timeout=timeout)
If I use requests, it works like a charm:
r = requests.post(url, files={tarFile: open(tarFile, 'rb')}, headers=headers)
I essentially need the equivalent of the above for urllib2. Does anybody maybe know it? I have checked the docs as well but I was not able to make it work..What am I missing?
Thanks!