How to save data in disk cache properly? - python

I want to save data between runs of my python application (data should be the same for different users). The best way I have found was save it in file on disk. Directory /var/cache/my_package looks good for this purposes, but I faced a few difficulty:
how to create this folder during the installation
how to set owner and access rules
how to access cache files without hardcoding absolute path
I have found data_file option in setuptools but it can't help with points 2 and 3.

diskcache does exactly what you are looking for!
Check out examples here: http://www.grantjenks.com/docs/diskcache/tutorial.html

Related

How to save program settings to computer?

I'm looking to store some individual settings to each user's computer. Things like preferences and a license key. From what I know, saving to the registry could be one possibility. However, that won't work on Mac.
One of the easy but not so proper techniques are just saving it to a settings.txt file and reading that on load.
Is there a proper way to save this kind of data? I'm hoping to use my wx app on Windows and Mac.
There is no proper way. Use whatever works best for your particular scenario. Some common ways for storing user data include:
Text files (e.g. Windows INI, cfg files)
binary files (sometimes compressed)
Windows registry
system environment variables
online profiles
There's nothing wrong with using text files. A lot of proper applications uses them exactly for the reason that they are easy to implement, and additionally human readable. The only thing you need to worry about is to make sure you have some form of error handling in place, in case the user decides to replace you config file content with some rubbish.
Take a look at Data Persistence on python docs. One option a you said could be persist them to a simple text file. Or you can save your data using some serialization format as pickle (see previous link) or json but it will be pretty ineficient if you have several keys and values or it will be too complex.
Also, you could save user preferences in an .ini file using python's ConfigParser module as show in this SO answer.
Finally, you can use a database like sqlite3 which is simpler to handle from your code in order to save and retrieve preferences.

can linux command line programs see python temporary files?

I have a simple web-server written using Python Twisted. Users can log in and use it to generate certain reports (pdf-format), specific to that user. The report is made by having a .tex template file where I replace certain content depending on user, including embedding user-specific graphs (.png or similar), then use the command line program pdflatex to generate the pdf.
Currently the graphs are saved in a tmp folder, and that path is then put into the .tex template before calling pdflatex. But this probably opens up a whole pile of problems when the number of users increases, so I want to use temporary files (tempfile module) instead of a real tmp folder. Is there any way I can make pdflatex see these temporary files? Or am I doing this the wrong way?
without any code it's hard to tell you how, but
Is there any way I can make pdflatex see these temporary files?
yes you can print the path to the temporary file by using a named temporary file:
>>> with tempfile.NamedTemporaryFile() as temp:
... print temp.name
...
/tmp/tmp7gjBHU
As commented you can use tempfile.NamedTemporaryFile. The problem is that this will be deleted once it is closed. That means you have to run pdflatex while the file is still referenced within python.
As an alternative way you could just save the picture with a randomly generated name. The tempfile is designed to allow you to create temporary files on various platforms in a consistent way. This is not what you need, since you'll always run the script on the same webserver I guess.
You could generate random file names using the uuid module:
import uuid
for i in xrange(3):
print(str(uuid.uuid4()))
The you save the pictures explictly using the random name and pass insert it into the tex-file.
After running pdflatex you explicitly have to delete the file, which is the drawback of that approach.

How to make file paths configurable within an ArcGIS Python Add-In?

This might be a general programming question but since I am doing it from within an Add-In therefore asking here at GIS forum. I have a project folder with sub-folders containing several files on my hard disk which I read from within my Python Add-In, its hard coded e.g.:
dem = r'C:/project/raster/dem'
and Add-In is in
r'C:/project/Add-In'
folder. I tried doing '../raster/dem' to define path of input raster layer but it failed to read. Please suggest how can I make it generic so that if I move project folder to D drive then Add-In would still be able to read data.
You can reference materials that are located within the add-in, everything that's included within the 'Install' directory will be copied into the appropriate location within AssemblyCache. You can then reference this by doing something like:
local_path = os.path.abspath(os.path.basename(__file__))
raster_path = os.path.join(local_path, 'rasters')
And from there, treat raster_path as a nomral path.
You could read in a configuration file stored under the user's profile. Because ArcGIS add-ins overwrite themselves every time the host application is opened, you don't want to store user-specific configuration information inside the add-in itself.
The configuration file can be in any format you want (e.g. XML, plain text), but the ConfigParser class makes reading and writing to an INI-like file format easy.

Python/Django: how to get files fastest (based on path and name)

My website users can upload image files, which then need to be found whenever they are to be displayed on a page (using src = ""). Currently, I put all images into one directory. What if there are many files - is it slow to find the right file? Are they indexed? Should I create subdirectories instead?
I use Python/Django. Everything is on webfaction.
The access time for an individual file are not affected by the quantity of files in the same directory.
running ls -l on a directory with more files in it will take longer of course. Same as viewing that directory in the file browser. Of course it might be easier to work with these images if you store them in a subdirectory defined by the user's name. But that just depends on what you are going to doing with them. There is no technical reason to do so.
Think about it like this. The full path to the image file (/srv/site/images/my_pony.jpg) is the actual address of the file. Your web server process looks there, and returns any data it finds or a 404 if there is nothing. What it doesn't do is list all the files in /srv/site/images and look through that list to see if it contains an item called my_pony.jpg.
If only for organizational purposes, and to help with system maintenance you should create subdirectories. Otherwise, there is very little chance you'll run into the maximum number of files that a directory can hold.
There is negligible performance implication for the web. For other applications though (file listing, ftp, backup, etc.) there may be consequences, but only if you reach a very large number of files.

Where does Flask store uploaded files?

Where does Flask store uploaded files before the application code has a chance to save the file? Unless I've missed something it doesn't appear to be showing up in the /tmp directory, which is what I'd have expected, and obviously it's not showing up in the directory I've specified in app.config['UPLOAD_DIRECTORY']. It's not storing it in memory, is it?
Did you check the documentation? It seems pretty clear:
So how exactly does Flask handle uploads? Well it will store them in the webserver’s memory if the files are reasonable small otherwise in a temporary location (as returned by tempfile.gettempdir())

Categories