I've found out how to load images from the file system into Django, but how do you get them out?
I've figured out how to get stuff from the database in my management command and can do a query like:
for m in my_models.objects.filter(get=some):
image = m.image
# How do I copy this to another non-django location?
Images (or other FileFields) are actually regular files stored in somewhere (like a file system or a block storage like S3). Assuming that your files reside on the server's file system, you can use the path property to get the original file path and use Python shell utilities to copy it to another location:
import shutil
for m in my_models.objects.filter(get=some):
image = m.image
shutil.copy(image.path, '/var/tmp/')
Related
In my project I have receiving multiple files using request.FILES.getlist('filedname') and saving it using django forms save method. Again reading the same files using tika server api of python:
def read_by_tika(self, path):
'''file reading using tika server'''
parsed = parser.from_file(str(path))
contents = (parsed["content"].encode('utf-8'))
return contents
Is there any way to directly put list files getting from request.FILES to tikka server without saving it on hard disk.
If the files are small, try using tika's .from_buffer() with file.read(). However, files over 2.5 MBs are anyway saved to temporary files by django, see Where uploaded data is stored. In this case use read_by_tika(file.temporary_file_path()). See also file upload settings
I have property model, containing a field image_url.
class Property(ndb.Model):
date_created=data.UTCDateTimeProperty(auto_now_add=True)
# some other fields here
image_url = ndb.StringProperty(indexed=False)
and image model,
class Image(ndb.Model):
property = ndb.KeyProperty()
file = ndb.KeyProperty(indexed=False)
# some other fields
image_url = ndb.StringProperty(indexed=False)
Now I have 'n' number of images for each property in my local machine. Name of each image is mapped to corresponding property id in csv file. I want to bulk upload all these images from my local machine to google app engine datastore or blobstore.
I tried to google up but feel like I am stuck, any help or reference would be highly appreciated.
Google Cloud Storage might be a better option for you:
You get a nice program to work with it, gsutil, that will let you upload easily from the console, so you can write your own scripts :)
You can keep the filenames you already have, and setup your own directory structure so that it makes more sense for your app. If data is static then you might not even need support models.
Example, from the links above, on how you'd end up uploading your images:
gsutil cp *.jpg gs://images
The cp command behaves much like the Unix cp command with the recursion (-R) option, allowing you to copy whole directories or just the contents of directories. gsutil also supports wildcards, which makes it easy for you to copy or move batches of files.
My goal was to duplicate my Google App Engine application. I created new application, and upload all needed code from source application(python). Then I uploaded previously created backup files from the Cloud Storage of the source application (first I downloaded those files to PC and than uploaded files to GCS bucket of the target app)
After that I tried to restore data from those files, by using "Import Backup Information" button.
Backup information file is founded and I can add it to the list of available backups. But when I try to do restore I receive error: "There was a problem kicking off the jobs. The error was: Backup not readable"
Also I tried to upload those files back to original application and I was able to restore from them, by using the same procedure, so the files are not corrupted.
I know there are another methods of copying data between applications, but I wanted to use this method. If for example, my Google account is being hacked and I can not access my original application data, but I have all backup data on my hard drive. Then I can simply create new app and copy all data to the new app...
Has anyone before encountered with the similar problem, and maybe found some solution?
Thanks!
Yes!! What you are trying to do is not possible. The reason is that there are absolute references in the backup files to the original backup location (bucket). So moving the files to another GCS location will not work.
Instead you have to leave the backup files in the original GCS bucket and give your new project read access to that folder. That is done in the "Edit bucket permissions" option. eg. add:
Project - owners-12345678 - Reader
Now you are able to import from that bucket in your new project in "Import Bucket Information".
Given the message, my guess is that the target application has no read access to the bucket where the backup is stores. Add the application to the permitted users to that bucket before creating the backup so that the backup objects will inherit the permission.
I would like to use binary fields in openerp, but instead of save them to the database as usual, I'd like to save to file system (folder). I could use a char field to store the path but, is there a way to implement the upload and download for it?
Thanks
A module which was developed in version 6 may help you to select path for saving files instead of storing in db as binary.
check out web gallery module
So I am trying to port a Python webapp written with Flask to Google App Engine. The app hosts user uploaded files up to 200mb in size, and for non-image files the original name of the file needs to be retained. To prevent filename conflicts, e.g. two people uploading stuff.zip, each containing completely different and unrelated contents, the app creates a UUID folder on the filesystem and stores the file within that, and serves them to users. Google App Engine's Cloud Storage, which I was planning on using to store the user files, by making a bucket - according to their documentation has "no notion of folders". What is the best way to go about getting this same functionality with their system?
The current method, just for demonstration:
# generates a new folder with a shortened UUID name to save files
# other than images to avoid filename conflicts
else:
# if there is a better way of doing this i'm not clever enough
# to figure it out
new_folder_name = shortuuid.uuid()[:9]
os.mkdir(
os.path.join(app.config['FILE_FOLDER'], new_folder_name))
file.save(
os.path.join(os.path.join(app.config['FILE_FOLDER'], new_folder_name), filename))
new_folder_path = os.path.join(
app.config['FILE_FOLDER'], new_folder_name)
return url_for('uploaded_file', new_folder_name=new_folder_name)
From the Google Cloud Storage Client Library Overview documentation:
GCS and "subdirectories"
Google Cloud Storage documentation refers to "subdirectories" and the GCS client library allows you to supply subdirectory delimiters when you create an object. However, GCS does not actually store the objects into any real subdirectory. Instead, the subdirectories are simply part of the object filename. For example, if I have a bucket my_bucket and store the file somewhere/over/the/rainbow.mp3, the file rainbow.mp3 is not really stored in the subdirectory somewhere/over/the/. It is actually a file named somewhere/over/the/rainbow.mp3. Understanding this is important for using listbucket filtering.
While Cloud Storage does not support subdirectories per se, it allows you to use subdirectory delimiters inside filenames. This basically means that the path to your file will still look exactly as if it was inside a subdirectory, even though it is not. This apparently should concern you only when you're iterating over the entire contents of the bucket.
From the Request URIs documentation:
URIs for Standard Requests
For most operations you can use either of the following URLs to access objects:
storage.googleapis.com/<bucket>/<object>
<bucket>.storage.googleapis.com/<object>
This means that the public URL for their example would be http://storage.googleapis.com/my_bucket/somewhere/over/the/rainbow.mp3. Their service would interpret this as bucket=my_bucket and object=somewhere/over/the/rainbow.mp3 (i.e. no notion of subdirectories, just an object name with embedded slashes in it); the browser however will just see the path /my_bucket/somewhere/over/the/rainbow.mp3 and will interpret it as if the filename is rainbow.mp3.