I am running a nextcloud instance and I am trying to download a directory with an API call using the library requests.
I can download a zip file using an API call. Now what I would like to to is to have an unzipped directory on my nextcloud instance and download it via an API call. It doesn't matter to me if I get a zip file back when I do the API call, I just want to have it unzipped in the cloud.
For instance, I can put an unzipped directory there and when I download it in my browser, nextcloud gives me back a zip file. This behaviour I want in an API call.
Now if I put a zipped file on there I can download the file like so:
import os
import requests
response = requests.request(
method="get",
url=f"https://mycloud/remote.php/dav/files/setup_user/{name_dir}/",
auth=("my_user", my_token),
)
if response.status_code == 200:
with open("/path/to/my/dir/file_name.zip"), "wb") as file:
file.write(response.content)
That writes me my zipfile which is in the cloud to a local file_name.zip file. My problem is now that if I have an unzipped directory in the cloud it doesn't work. Doesn't work meaning that I get a file back which has the content:
This is the WebDAV interface. It can only be accessed by WebDAV clients such as the Nextcloud desktop sync client.
I also tried to do this with wget wget --recursive --no-parent https://path/to/my/dir and I got the same file with the same content back.
So I assume that the WebDav API of nextcloud doesn't allow me to do it in the way I want to do it. Not I am wondering what I am doing wrong or if what I want is doable. Also I don't get why in the browser this works fine. I just select the unzipped folder and can download it with a click. In the nextcoud community it has been suggested to use Rclone (https://help.nextcloud.com/t/download-complete-directory-from-nextcloud-instance/77828), but I would prefer to not use a dependency that I have to set up on every machine where I want to run this code.
Any help is very much appreciated! Thanks a bunch in advance!
PS: In case anyone wonders why I would want to do this: It's way more convenient when I would like to change just a single file in my dir in the cloud. Otherwise I have to unzip, change file, zip and upload again.
Related
Can i use the url to a google drive folder as a download_path in selenium
for instance like this .Heroku error says file not found
mudopy.download_path(r"https://drive.google.com/drive/folders/1IPNuefeIXXCKm8Xxm3u1ekltxTF3dCT0")
According to the code of mudopy module, it saves the downloaded file into a local directory. So you can later upload that file to a google drive. Not sure if heroku allows to store something locally.
Moreover, the function mudopy.download_path does not download anything yet, but only creates a local file that stores the path to the folder where you want to save the downloaded files. And looks like Heroku does't even permit creating this file.
I have a working code that sends a GET request to a server and downloads files to my local folder. Now I need to have these files available on a SharePoint. I can upload the files to SharePoint after they are downloaded to my local folder. But is there a way that I can run my GET request and directly download the file to SharePoint? I am okay with python or node.
I have a dataset contains hundreds of numpy arrays looks like this,
I am trying to save them to an online drive so that I can run the code with this dataset remotely from a sever. I cannot access the drive of the server but can only run code script and access the terminal. So I have tried with google drive and Onedrive, and looked up how to generate a direct download link from those drives but it did not work.
In short, I need to be able to get those files from my python scripts. Could anyone give some hints?
You can get the download URLs very easily from Drive. I assume that you already uploaded the files into a Drive folder. Then you can easily set up a scenario to download the files on Python. First you would need an environment on Python to connect to Drive. If you don't currently have one, you can follow this guide. That guide will install the required libraries, credentials and run a sample script. Once you can run the sample script you can make minor modifications to reach your goal.
To download the files you are going to need their ids. I am assuming that you already know them, but if you don't you could retrieve them by doing a Files.list on the folder where you keep the files. To do so you can use '{ FOLDER ID }' in parents as the q parameter.
To download the files you only have to run a Files.get request by providing the file id. You will find the download URL on the webContentLink property. Feel free to leave a comment if you need further clarifications.
I have a flask app in which it has a button to download an HTML report. The download button when hit creates an xlsx file. It worked fine until that app was running on my local server because the python code found my Downloads directory no matter what os it is using python's os module.
Now the app is deployed on a remote server, in this case, how do you let the system know the client's download directory path where this xlsx file can then be created? Or any other pointers for a better solution if I'm missing something?
To serve a user a file to download, the webserver needs to deliver it in a special way. You can't use the os module to write to the user's computer. It worked locally, because your server was the same computer as your user environment in the browser.
The correct way to serve a user a file for downloading is to use flask's send_file method.
To show you the process, I've created an example below. In this example, I'm assuming that you have a function called build_report(...) which accepts a report_id and returns an io.BytesIO object containing the content of the xlsx file.
from flask import send_file
#app.route("/download-report/<report_id>")
def download_report(report_id):
file_obj = build_report(report_id)
return send_file(
file_obj,
mimetype="application/vnd.ms-excel",
attachment_filename=f"report-{report_id}.xlsx",
)
If i understand correctly - you want to specify a directory to which the file should be downloaded on users computer when he/she hits download button.
This is not possible, and is handled fully by the browser.
Browser processes the request/stream of special type and then creates the output file in the location specified by the user in browser settings. The os library which you mentioned relates to your server machine not client, so any os opertaions that you provide in your code will be executed on your server (like creating a file). So that's why it worked on your local machine - which was server and client at once.
Why it's dissalowed?
Imagine a "virus file" being uploaded to your C:\Windows\System32. Web applications could be granted control over your machine with a simple button download. The huge security issue doesnt allow for client's machine access from web application
I tried using gsutil to download files in a bucket, but now would like to incorporate the download in a python script to automate the download process (for downloading specific days data). The following gsutil code worked fine.
gsutil -m cp -r gs://gcp-public-data-goes-16/GLM-L2-LCFA/2019/001 C:\dloadFiles
Using the storage client I have tried:
from google.cloud import storage
client = storage.Client()
with open('C:\dloadFiles') as file_obj:
client.download_blob_to_file(
'gs://gcp-public-data-goes-16/GLM-L2-LCFA/2019/001', file_obj)`
I get error "DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started"
This is a publicly available bucket.
You did not setup GOOGLE_APPLICATION_CREDENTIALS
Follow below link and setup credentials
https://stackoverflow.com/questions/45501082/set-google-application-credentials-in-python-project-to-use-google-api
After setting up credentials your code will work
After authenticating with your GCP credentials, you will also need to run:
gcloud auth application-default
To authenticate your application SDKs, such as your Python client libraries. Then you will be able to interact with GCP services via Python.
Also, you are copying a whole load of files with your gsutil command and not just one as you're doing with python. So you probably want to list_blobs first and then iteratively download them to files.
Also check out blob.download_to_file save you some coding (docs here). With that you can send a blob to a filename directly, without opening the file first.
First thing, turn off public on this bucket unless you really need the bucket to be public. For private access, you should use a service account (your code) or OAuth credentials.
If you are running this code in a Google Compute Service, credentials will be automatically discovered (ADC).
If you are running outside of Google Cloud, change this line:
client = storage.Client()
To this:
client = storage.Client().from_service_account_json('/full/path/to/service-account.json')
This line in your code is trying to open a directory. This is not correct. You need to specify a file name and not a directory name. You also need to specify write permission:
with open('C:\dloadFiles') as file_obj:
Change to
with open('c:/directory/myfilename', 'w')
Or for binary (data) files:
with open('c:/directory/myfilename', 'wb')
I am assuming that this path is a file blob and not a "simulated" folder on GCS. If this is a folder, you will need to change it to a file (storage object blob).
gs://gcp-public-data-goes-16/GLM-L2-LCFA/2019/001