I have a deployed django webapp which has been working fine. It scrapes sites for newly published .csv files and then uses them for data analysis which is then returned to the user.
The most recent .csv is not being recognised as a file on the deployed version, but is on the test version on my local machine. The structure is as follows:
-indicator-analyser
-Analyser
-AnalysisScripts
-uploads
-data
-2017
-Monthly_File_Jun_Final.csv
-Monthly_File_Sep_Final.csv
When a user attempts to run the script on the Monthly_File_Jun_Final.csv, the webapp performs as expected. When they run the same script on Monthly_File_Sep_Final.csv, django throws an error as there is no file found. I have taken the file path that is passed in and used that to open the file in explorer, and I have used the same file path to load the .csv as a dataframe in pandas within the console with no problems.
The path that is passed to the loading script is:
C:\\webapp\\indicator-analyser\\Analyser/uploads/data/2017/Monthly_File_Sep_Final.csv
When this is evaluated using os.path.isfile(filepath), it is being returned as False. However, when the other file is selected, this is returned and recognised as a file:
C:\\webapp\\indicator-analyser\\Analyser/uploads/data/2017/Monthly_File_Jun_Final.csv
Just for reference, this is running on a IIS server. I have restarted the machine and the server to no avail.
To conclude, I can access this file:
through the console
on my local instance
through my file explorer
But it is not recognised as a file in the live django instance.
This was fixed by deleting the scraped file and redownloading it, placing it in the exact same place. I do not know why this has fixed it, but it is now working.
Related
Can i use the url to a google drive folder as a download_path in selenium
for instance like this .Heroku error says file not found
mudopy.download_path(r"https://drive.google.com/drive/folders/1IPNuefeIXXCKm8Xxm3u1ekltxTF3dCT0")
According to the code of mudopy module, it saves the downloaded file into a local directory. So you can later upload that file to a google drive. Not sure if heroku allows to store something locally.
Moreover, the function mudopy.download_path does not download anything yet, but only creates a local file that stores the path to the folder where you want to save the downloaded files. And looks like Heroku does't even permit creating this file.
I have a Python script that scrapes data from a webpage.
It works on my local but not on AWS Lambda because it only allows writes new file to /tmp directory.
I tried to go through request_html render API and it seems like it's not possible to change the file location. It default take current working directory by default. It's causing error as below
Error detail Read-only file system: '/home/sbx_user123'
Thank you in advance.
How's set an environment variable below from lambda web UI?
PYPPETEER_HOME=/tmp/
It may change working directory of pyppeteer (a module running inside request_html).
maybe, it's a related question: Pyppeteer fails to download headless chrome when running on AWS Lambda
As the title suggests, I'm looking for a way to run local python scripts from the Google Apps Scripts environment. Right now my method is to write a json file from app scripts and save it to a Google Drive directory. The local machine polls the directory via Google Drive File Stream, and runs the necessary code when the file appears in the directory. Not pretty, but it gets the job done. The main concern with this is that File Stream can be fairly latent - could be up to 15 minutes after a file is put in the directory before File Stream catches it and syncs the local machine.
Is anybody aware of other options? I could try re-writing the python code in Apps Script, though the python code relies on a few third party libraries that aren't available in JS. Another idea is setting up a flask server, but that seems like a good bit of work for the return.
I am working on a cookiecutter-django project that requires users to be able to upload files in the .5-1gb size range. From what I understand when uploading files that are larger then 2.5Mbs django uses the TemporaryFileUploadHandler to first upload the file to the tmp directory and then to then transfer them to the media directory. However when I try to upload a larger file my site just hangs, I don't see any files come into the tmp directory, and I see no errors in the logs.
Is there some setting I need to turn on to instruct django to use the TemporaryFileUploadHandler? Or is there some way to possibly figure out where the site might be getting hung up? I am currently building out this project locally so I have full control over my environment.
I am using Python 3.4.5 and django 1.10
UPDATE 1
I have tested uploading files of 5mb, 10mb, 20mb, and 30mb they all upload fine. The file I am having issues with is 873mb.
UPDATE 2
After more debugging I have found that when up loading a large file to my site it gets as far as the get_form method on my view class. After that I am not sure what is called. I have been slowly adding logs at each step in the hopes of figuring out where it is stuck.
Just installed Google Apps Engine and am getting "could not save" errors.
Specifically if I go in to preferences I get
Could not save into preference file
C:\Usera\myname/Google\google_appengine_launcher.ini:No such file or directory.
So some how I have a weird path, would like to know where and how to change this. I have search but found nothing, I have done a repair reinstall of GAE
Can find nothing in the registry for google_appengine_launcher.ini
I first saw the error when I created my first Application
Called hellowd
Parent Directory: C:\Users\myname\workspace
Runtime 2.7 (PATH has this path)
Port 8080
Admin port 8080
click create
Error:
Could not save into project file
C:\Users\myname/Google\google_appengine_launcher.ini:No such file or directory.
Thanks
I think I have found the answer to my own question.
I have a small app I have written to backup my stuff to Google Drive, this app would appear to have an error in it that does not stop it from running but does cause it to make a file called
C:\Usera\myname\Google
Therefore GAE can not create a directory called C:\Usera\myname/Google nor a file called C:\Usera\myname/Google\google_appengine_launcher.ini
I deleted the file Google, made a directory called Google and ran the GAE, saved pereferences and all working