I need to store dynamic (will change every day/hour) value in django app.
Value will be generated every day, and I will need access to this key:value from all views in the application.
I don't want to store it in some model object instance, it should work like settings.py but I need to store it in database instead.
Any ideas?
what about leveldb? https://code.google.com/p/leveldb/
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
there is also python wrap https://code.google.com/p/py-leveldb/
or if you need distributed access checkout http://memcached.org/
Why not use a key-value datastore like Redis? You can use it as a cache backend or as a normal store, it's fast and will persist your data. https://django-redis.readthedocs.org/en/latest/ https://pypi.python.org/pypi/redis/2.10.1
Related
I'm in the process of learning Django, and I'm building an app that has a "User" model, and I need to store a dictionary of private API keys for various other applications for each user, where the key is the service name and the value is the actual API key. My current plan was to have the dictionary be stored as JSON, but I was wondering if there is a better/more secure way I should be doing this?
1) Use a extra Model with 1to1 relation to the default Django User Model (extendable, nicer)
2) Use a hashfield to secure the data
See https://github.com/amcat/django-hash-field
djangomachine is right. Hashing is not the answer because you need to use the actual key at some point.
You need to encrypt the key upon storage and decrypt the key upon retrieval. Look into https://github.com/incuna/django-pgcrypto-fields for example.
I am interested in adding a spell checker to my app -- I'm planning on using difflib with a custom word list that's ~147kB large (13,025 words).
When testing user queries against this list, would it make more sense to:
load the dictionary into memcache (I guess from the datastore?) and keep it in memcache or
build an index for the Search API and pass in the user query against it there
I guess what I'm asking is which is faster: memcache or a search index?
Thanks.
Memcache is definitely faster.
Another important consideration is cost. Memcache API calls are free, while Search API calls have their own quota and pricing.
By the way, you may store your library as a static file, because it's small and it does not change. There is no need to store it in the Datastore.
Memcache is faster however you need to consider the following.
it is not reliable, at any moment entities can be purged. So your code needs a fallback for non cached data
You can only fetch by key, so as you said you would need to store whole dictionaries in memcache objects.
Each memcache entity can only store 1MB. If you dictionary is larger you would have to span multiple entities. Ok not relevant in your case.
There are some other alternatives. How often will the dictionary be updated ?
Here is one alternate strategy.
You could store it in the filesystem (requires app updates) or GCS if you want to update the dictionary outside of app updates. Then you can load the dictionary in each instance into memory at startup or on first request and cache it at the running instance level, then you won't have any round trips to services adding latencies. This will also be simpler code wise (ie no fallbacks if not in memcache etc)
Here is an example. In this case the code lives in a module, which is imported as required. I am using a yaml file for additional configuration, it could just as easily json load a dictionary, or you could define a python dictionary in the module.
_modsettings = {}
def loadSettings(settings='settings.yaml'):
x= _modsettings
if not x:
try:
_modsettings.update(load(open(settings,'r').read()))
except IOError:
pass
return _modsettings
settings = loadSettings()
Then whenever I want the settings dictionary my code just refers to mymodule.settings.
By importing this module during a warmup request you won't get a race condition, or have to import/parse the dictionary during a user facing request. You can put in more error traps as appropriate ;-)
Is it possible to represent data in Google App Engine's datastore view differently in Python?
For example, if I create a list of ndb.KeyProperty(repeated=True) when I look in the datastore it shows up as: [datastore_types.Key.from_path(u'User', 6544293208522752L, _app=u'dev~appstuffhere')]
Is there any way to change the representation of that to something else? I was working with __str__, __unicode__, and __repr__ to see if that is what App Engine looks to for representing this data, but to no avail.
I would prefer to see a list of User Names rather than that datastore_types.Key.from_path representation.
Is this possible?
The datastore viewer doesn't know about your models at all. It purely goes on the entity kinds in the datastore, which are independent of the models themselves.
As Paul says in the comment, if you need a different representation you'll need to build it yourself. To get the usernames from the key properties you'd need to actually fetch the data (remember, the datastore is not a relational db, so there's no such thing as a join):
usernames = ', '.join(e.username for e in ndb.get_multi(e.users))
I am writing function, which create new Solr core.
To create core, you need to post data like (http://wiki.apache.org/solr/CoreAdmin):
http://localhost:8983/solr/admin/cores?action=CREATE&name=coreX&instanceDir=path_to_instance_directory&config=config_file_name.xml&schema=schem_file_name.xml&dataDir=data
But in this example you need to refer to existing config and schema.
In my app each core can be with different configuration, so the best way will be to post config and schema with JSON format to server with create request.
Its is possible?
Thanks for the help!
No, as far as I know, this is not possible at the moment (without creating the files on the server, then creating a core from the files).
You might want to use a more schemaless-ish structure for your schema if you need this kind of functionality, where you rather define a set of field pre/postfixes that map to different default settings for fields, and then use wildcard names to avoid having to define each field in your Schema.
A truly schema less alternative based on Lucene could be Elastic Search as well.
I want to use all the data in my python app engine memcache. I do not know the keys in advance.
How do I go about getting all data?
The only read functions available on memcache are:
get(key, namespace=None)
get_multi(keys, key_prefix='', namespace=None)
As you can see, to get data from memcache you must provide one or more keys.
I am using a 'well known key' called "config" where I store a list of all other keys and use that to enumerate the rest of the items.
as in comments above, I guess I could stick all data in a single memcache entry with a known key.
Still for non-static data there are scenarios where it would be useful.