So the issue I am facing is trying to get our webservers export to CSV button working, one requirement is the file cannot be saved locally and must be streamed straight to the client. At the moment I am just trying to get a basic CSV file streamed and saved on the local machine from the server before I worry about rendering actual data.
So, I have, on button click a post to my views where I wish to render then return a response. This was a $.ajax function but I have since changed it during testing and it now looks like this looks like:
$("#exportbutton").click(function(e){
$.post("{% url 'doner:download' %}", function(data)
{
console.log(data);
});
})
The code in the view looks like this (which happens to be a CTRL+V of an example from SO):
def download(request):
def stream():
buffer_ = io.StringIO()
writer = csv.writer(buffer_)
for row in rows:
writer.writerow(row)
buffer_.seek(0)
data = buffer_.read()
buffer_.seek(0)
buffer_.truncate()
yield data
response = StreamingHttpResponse(
stream(), content_type='text/csv'
)
disposition = "attachment; filename=file.csv"
response['Content-Disposition'] = disposition
return response
So after a lot of tinkering this is as far as I have got, it will 'stream'(?) the data back into the HTML page, the console.log will output the 'files' data however I do not get any form of a file or a download, just the log in the console of the data.
I don't know if I have missunderstood how StreamingHttpResponse works and I'm meant to do something more on the client side in order to download it past the stream but I've read the docs and I'm no more clear.
Any help would be greatly appreciated as I'm stumped, cheers guys!
The issue I was facing was posting using jQuery, apparently the browser is not able to download a file using AJAX, so in order to fix this part and get the file download, all I did was inside the AJAX (making sure it get generated alright before it's downloaded) add the line location.replace('doner/download'); and that was it...
Now the jQuery looks like this:
$("#exportbutton").click(function(e){
$.post("{% url 'doner:download' %}", function(data)
{
console.log(data);
location.replace("{% url 'doner:download' %}");
});
})
And everything works as needed!
Related
I have almost the question already posted and answered here:
Perform Download via download button in Python
I also followed the instructions, in the answer of the above link.
In my case I want to download the data from the following page:
https://www.smard.de/home/downloadcenter/download-marktdaten#!?downloadAttributes=%7B%22selectedCategory%22:1,%22selectedSubCategory%22:1,%22selectedRegion%22:%22DE%22,%22from%22:1658872800000,%22to%22:1659563999999,%22selectedFileType%22:%22CSV%22%7D
The download-button is the one called "Datei herunterladen" in the lower right corner.
I went into the inspect mode downloaded the file and got the following output in the inspection
But the resulting
Request URL: 'https://www.smard.de/nip-download-manager/nip/download/market-data'
does not help getting the csv-file. Opened in the browser I get: 'The requested URL was rejected.'
On the other hand, it does not even contain the parameters anymore, so it can't be the right download url.
May anyone help to automate this download?
edit
Now I also tried
url = 'https://www.smard.de/nip-download-manager/nip/download/market-data'
json_body = {'format': "CSV",
'language': "de",
'moduleIds': [1001224, 1004066, 1004067, 1004068, 1001223, 1004069, 1004071, 1004070, 1001226, 1001228, 1001227,1001225],
'region': "DE",
'timestamp_from': 1659304800000,
'timestamp_to': 1659391199999,
'type': "discrete"}
x = requests.post(url, json = json_body)
x.content
> b'Exception when parsing incoming request to JSON object.'
So how do I get the csv-file based on this method?
the reason that you're getting "URL Rejected" is because the link specified in the request is not file-specific. The link that the POST request is being made to, contains several files, which are given out based on the body that the request is sent with. If you look into the developer tools, you can see that there is a request payload attached to the request (image attached) . If you want to learn more about sending JSON body data with requests, please take a look at w3 and this thread. I hope that helps!
The json_body of the request-payload was not correct.
And I found an easy way to get the correct one with less effort and less probability to make it wrong.
In the inspect mode after the file is downloaded:
Click on the "Raw"-Button in the upper right corner of the "request"-tab, than you can get the correct payload per copy and paste.
The final solution is:
url = 'https://www.smard.de/nip-download-manager/nip/download/market-data'
payload = {"request_form":[{"format":"CSV","moduleIds":[1001224,1004066,1004067,1004068,1001223,1004069,1004071,1004070,1001226,1001228,1001227,1001225],"region":"DE","timestamp_from":1658872800000,"timestamp_to":1659563999999,"type":"discrete","language":"de"}]}
x = requests.post(url, json = payload)
with open(f'energy_ger.csv', 'wb') as f:
f.write(x.content)
I am using a scraper to upload json data to a database but I noticed that my uploads are not in order yet my local json data file FILE_PATH mirroring the database has my json data in order, meaning that the culprit is in how I use python's request module.
def main():
if not FILE_PATH.is_file():
print("initializing json database file")
# intialize and reverse the dictionary list in one go, [start:stop:step]
ref_links = getArchivedCharacterList()[::-1]
index =0 # counting up the roster size for assignReleasedate
for link in ref_links:
char_details = getCharacterDetails(link['ref_id'],link['game_origin'],index)
index +=1
saveCharacterInfo(char_details)
postRequest(POST_CHARA_DETAILS_URL,char_details,headers)
print(f"Done! Updated the file with all previously released and archived units as of: \n {datetime.datetime.now()} \n See {FILE_PATH} for details!")
From the above, I initially scrape a page for a list of links using getArchivedCharacterList(), then for each link, I grab more information for individual pages and then I post them into my local file FILE_PATH using saveCharacterInfo and I POST it into my database with the postRequest function.
postRequest function
def postRequest(url,json_details,headers):
r = requests.post(url, json=json_details, headers=headers)
while r.status_code != 201:
time.sleep(0.1)
print(r.status_code)
I tried doing a while function to wait for the 201 POST success response. Didn't work. I looked up async/await tutorials and it doesn't seem like something I want... unless I am supposed to bundle up all my request posts in a single go? I did this before in javascript where I bundled up all my posts in a promise array is there a promise equivalent for python from javascript so that I do my uploads in order? Or is there another method to achieve a sequential upload?
Thank you in advance!
I got an API called remove.bg . I want to use this API ( provided in python language ) in my Flutter App. Is it even possible?
This API uses for removing image background.
What are the steps/ research I need to do to get this thing working?
Do lots of Googling, but ends up with nothing.
Really Appreciate your help!!!
OR can I use this link and able to upload and get the output in my app?
for example, I open the APP, and it will show two-button -> Upload image & download image.
when user clicks the Upload button it will redirect to this link and after processing done in the website, the output we can able to download in our app.
This is possible with Flutter's http package. Assuming it is some form of RESTful API this should give you a starting point:
final body = {"image_file": "#/path/to/file.jpg", "size": "auto"};
final headers = {"X-API-Key": INSERT_YOUR_API_KEY_HERE};
final response = await http.post('https://api.remove.bg/v1.0/removebg',
body: body,
headers: headers);
if (response.statusCode == 200) {
// do something with response.body
} else {
throw Exception('Failed to do network requests: Error Code: ${response.statusCode}\nBody: ${response.body}');
}
A good tutorial on http in Flutter is here.
Note: You may have to do json.encode(body) and the same with header and use json.decode(response.body) depending on the API.
Hope it helps, and if so please up vote and accept as answer and if not please leave a comment below.
I am communicating with an API using HTTP.client in Python 3.6.2.
In order to upload a file it requires a three stage process.
I have managed to talk successfully using POST methods and the server returns data as I expect.
However, the stage that requires the actual file to be uploaded is a PUT method - and I cannot figure out how to syntax the code to include a pointer to the actual file on my storage - the file is an mp4 video file.
Here is a snippet of the code with my noob annotations :)
#define connection as HTTPS and define URL
uploadstep2 = http.client.HTTPSConnection("grabyo-prod.s3-accelerate.amazonaws.com")
#define headers
headers = {
'accept': "application/json",
'content-type': "application/x-www-form-urlencoded"
}
#define the structure of the request and send it.
#Here it is a PUT request to the unique URL as defined above with the correct file and headers.
uploadstep2.request("PUT", myUniqueUploadUrl, body="C:\Test.mp4", headers=headers)
#get the response from the server
uploadstep2response = uploadstep2.getresponse()
#read the data from the response and put to a usable variable
step2responsedata = uploadstep2response.read()
The response I am getting back at this stage is an
"Error 400 Bad Request - Could not obtain the file information."
I am certain this relates to the body="C:\Test.mp4" section of the code.
Can you please advise how I can correctly reference a file within the PUT method?
Thanks in advance
uploadstep2.request("PUT", myUniqueUploadUrl, body="C:\Test.mp4", headers=headers)
will put the actual string "C:\Test.mp4" in the body of your request, not the content of the file named "C:\Test.mp4" as you expect.
You need to open the file, read it's content then pass it as body. Or to stream it, but AFAIK http.client does not support that, and since your file seems to be a video, it is potentially huge and will use plenty of RAM for no good reason.
My suggestion would be to use requests, which is a way better lib to do this kind of things:
import requests
with open(r'C:\Test.mp4'), 'rb') as finput:
response = requests.put('https://grabyo-prod.s3-accelerate.amazonaws.com/youruploadpath', data=finput)
print(response.json())
I do not know if it is useful for you, but you can try to send a POST request with requests module :
import requests
url = ""
data = {'title':'metadata','timeDuration':120}
mp3_f = open('/path/your_file.mp3', 'rb')
files = {'messageFile': mp3_f}
req = requests.post(url, files=files, json=data)
print (req.status_code)
print (req.content)
Hope it helps .
I've a pdf file in one of my main project directory. How should I add this file in the main urls.py file so as to link this file in <a href> tag.
EDIT
I get 2 dates, start and end date, via AJAX. I process the data b/w those 2 dates, generate a report and then returns an HttpResponse. The PDF report is now saved in my main project directory. Now I get a response back in AJAX. So, now how should I process the response in the success function, sent back from the sever and open a PDF file.
Thanks.
jQuery
$(function() {
$("#report_submit").click(function(){
$.ajax({
type : "POST",
url: "/reports/",
data : { 'start_date' : $("#startDate").val() , 'end_date' : $("#endDate").val() },
success : function(result){
},
error : function(result){
}
});
});
});
Django view code
def generate_report(request):
ctx = {}
if request.is_ajax():
if request.POST.has_key('start_date'):
start_date = datetime.strptime(request.POST[ 'start_date'] , '%m/%d/%Y')
end_date = datetime.strptime(request.POST[ 'end_date'] , '%m/%d/%Y')
......
# PDF GENERATED in MAIN PROJECT DIRECTORY
with open(os.path.join(os.path.dirname(__file__),'../../../../gui','Report.pdf')) as pdf:
response = HttpResponse(pdf.read(), content_type='application/pdf')
response['Content-Disposition'] = 'inline;filename=Report.pdf'
return response # so, now when I send a response back, how should I process it in AJAX success function?
pdf.closed
return render(request, 'generate_report/reports.html', ctx)
You can do this in two ways depending upon the requirements of your app
Add the pdf file to the static directory and serve it from there. After all pdf file is really just a static resource.
However, in case you need to do some checks before serving the file to the user eg. allowing only the authenticated users to access it, then write a view that will do the necessary checking and then serve the contents of the file with appropriate response headers. See example here
Edit after the OP updated their question
Is ajax absolutely necessary here? In case it is I can think of following:
Since you mention a report is being generated, I will assume that non authenticated and no authorized users shouldn't be able to access the report.
One way is to store the generated pdf file at some location (outside the static dir) and also save a record of the name of the generated file and the id of the user who can access it in a database table. The request can then simply respond with the name of the file.
In the success callback of ajax, open the url of a view along with the filename as get param in a separate tab. This view will check if the user can access the file and serve it.
Again, all this complexity can be avoided if ajax is not a requirement